Some clarifications below:
Thilo Goetz wrote:
Marshall Schor wrote:
Using the definitions Adam defined:
* "CAS" means the entire CAS. It never means a specific view of the
CAS.
* "Index Definition" means the declaration in the descriptor that
defines an index - giving it a label, kind of index, CAS type, and
sort keys.
* "Index" is an instance of an index definition - something that can
be retreived by a getIndex() call and from which you can get an
iterator.
* "Physical Index" is an actual data structure holding references to
FeatureStructures. This is transparent to the user but sometimes we
need to talk about it if we're concerned about performance.
To this, let me add:
* "Index Set" - a collection of Index definition instances - or
Indexes (for short) -
identified by a name (called the "view name").
I'm not sure this wasn't settled in your discussion with Adam, but to
my current way of thinking, a non-anchored view is nothing but a named
set of indexes. So this definition of an index set seems redundant.
OK. I was trying to keep the concepts simpler, more circumscribed:
CAS - a container
- can have a Sofa, can have more than 1 Sofa
- has an Index Set, can have more than 1 Index Set
Special case: "Anchored View": An Index set that has 1 associated Sofa
With this formulation, you can see that there are other potential
combinations:
non-Anchored view: An Index Set without an associated Sofa
multi-Anchored view (?? I made up this name, not really suggesting
it ??): An Index set with more than 1 associated Sofa.
<snip>
3) A magic method exists for tools to get all the FS's out of a CAS
(when serializing).
- This magic method can be restricted to just those FS's that are
indexed in some index,
or which is reachable from a chain of references starting in
another FS which is indexed.
I'm not quite sure what you mean here, but if this implies that this
magic method can also return FSs that are not indexed anywhere, I
don't think so.
That's OK - which is why I said the 2nd sentence.
FSs that are not indexed are meant to be temporary and local to an
annotator, so no need to serialize them or do anything else with them.
Data that is *local* to an annotator is most likely never put into the
CAS, but would rather held in "native" annotator data structures,
because (a) it's usually more efficient, and (b) the space is reclaimed
(of course, depending on the Annotator design, and assuming we aren't
adding garbage-collection to the CAS).
It seems to me a more convincing use-case for this is data that was put
into the CAS (to be shared) which some
subsequent process effectively "deleted" (e.g., changed a reference that
was serving to locate the FS).
Can we stop there (here)? I think with these concepts we can build
the higher level
concepts we now have, efficiently, except for the concept of
subsetting the FS's by "index-set".
Currently, we don't have a way to define an index which is a "filter"
- including some members
of a type, while excluding others. An abstract example: "odd-token"
and "even-token" - both
being "token" types, but one only holding the "odd" ones, etc. As
Thilo has pointed out -
the index could contain all token types, and a "filtered-iterater"
could be used at iteration time
to sort these out, as an alternative. There are of course space/time
tradeoffs here.
- If we did have a way to define an index which is a "filter", we
might be able to
efficiently use this to do the same thing that index-sets enable,
perhaps in a more general
way.
More general how?
Multiple Index sets are an additional mechanism (compared with not
having multiple index sets). They provide
a way to say a FS is a "member" of some index set.
A more general approach (one which doesn't add any new mechanisms to
having a single Index Set) would be to
get rid of multiple index sets, and say if users want to make FSs
"members" of some user-defined "sets" (called
views), they can do that using normal indexes (assuming we have added
the ability to define indexes which
filter). Here's how it could work:
- Simple case: User desires FSs to be members of 1 view:
- User defines additional structures to support the way they want
to refer to these.
- e.g. an additional slot per FS, with a ref to a "view"
object they define.
- User defines an index over the type they want in the view,
with the extra predicate that
slot value == the view object
This may not be such a good idea because it requires additional slot /
FS, and it requires some "management" of
view names to be able to specify the equal test.
So - I think I would come down in favor of having multiple named index
sets as a better approach here.
Just think of this thread as an intellectual exploration of
possibilities, not as something I'm advocating :-)
- Otherwise, we could use the concept of index-set to specify this
filter:
4) FS's can be indexed (but don't have to be).
- If there is more than one "index set", you have to specify which
"index set" to use;
the index operations (add/remove) update only the indexes in
that index-set.
(Note that this doesn't fit with other ideas where a particular
"index" might be in
multiple index sets. In this proposal, the only way to put an
instance into multiple
index sets is to do multiple adds, one per index-set.)
That's what we have views for, isn't it?
Right. I think these are equivalent. A view corresponds to a (named)
index-set.
This doesn't have the concept of "global indexes". If you want that,
you can create another
"index set" and use it for that purpose.
<snip>
The set of all indexes must be accessible to the user in the CAS,
otherwise we violate the "all data must be accessible from the CAS
without recourse to views" constraint.
I think this is satisfied by my so-called "magic method". This would
likely be implemented as suggested above.
Most *typical users* probably would not use APIs which iterate through
all indexes, but framework & tooling would.
-Marshall
