Re: [DISCUSS] TinkerGraph Memory Usage Enhancements

[Stephen Mallette]

could you please talk about what made you heat back up on the idea?

On Sun, Feb 11, 2018 at 6:06 AM, Robert Dale  wrote:

> I think this 'schema mode' is a good addition.
>
> Robert Dale
>
> On Sun, Jan 14, 2018 at 3:41 AM, Michael Pollmeier <
> mich...@michaelpollmeier.com> wrote:
>
> > First of all: if you're not manually enabling the memory-efficient mode,
> > you get the exact same behaviour as now.
> >
> > With the memory-efficient mode enabled, a few semantics change (by
> design):
> > * setting a property that is not schema-conform may (IMO *should*, but
> > that depends on the implementation) throw an error
> > * generic properties (and meta properties) change conceptually, but you
> > can still have them
> >
> > There's no changes to the underlying queries necessary. We didn't have
> > to adapt ours, anyway.
> >
> > On 13/01/18 11:34, Robert Dale wrote:
> > > Michael, are there any limitations other considerations? That is, are
> > there
> > > any graph features that are lost or schema doesn't apply to, e.g.
> > > meta-properties?
> > >
> > > Robert Dale
> > >
> > > On Thu, Jan 11, 2018 at 4:03 PM, Michael Pollmeier <
> > > mich...@michaelpollmeier.com> wrote:
> > >
> > >> Thanks Stephen. Just adding a few notes:
> > >>
> > >> * As mentioned before I'm happy to provide a PR to the Apache
> > >> Tinkergraph and thereby close down our fork. If others prefer to not
> > >> dilute the waters, I'll equally happily maintain the fork and bring in
> > >> changes from Tinkerpop releases (as I've just done with 3.3.1)
> > >>
> > >> * By default, even our fork still uses the same (generic, non-strict)
> > >> mode. The user can enable the specialised/strict/memory-efficient
> mode
> > >> by providing factories for their specialised elements
> > >>
> > >> * It's all-or-nothing: either all elements are specialised or all
> > >> elements are generic. Technically this can be changed, I just didn't
> > >> have the need and time to do it.
> > >>
> > >> Cheers
> > >> Michael
> > >>
> > >>
> > >>
> > >> On 12/01/18 00:50, Stephen Mallette wrote:
> > >>> Michael Pollmeier recently authored a blog post that described how
> > their
> > >>> fork of TinkerGraph memory usage could be reduced by 70% assuming
> usage
> > >> of
> > >>> a strict schema:
> > >>>
> > >>> https://blog.shiftleft.io/open-sourcing-our-specialized-
> > >> tinkergraph-with-70-memory-reduction-and-strict-schema-
> > >> validation-fa5cfb3dd82d
> > >>>
> > >>> The question at this point, is whether or not similar enhancements
> > should
> > >>> be made to TinkerPop's TinkerGraph. I've had a few minutes to get to
> > >>> understand the changes that make the memory improvement possible -
> > here's
> > >>> my thoughts:
> > >>>
> > >>> 1. This was a clever way to extend TinkerGraph.
> > >>> 2. The schema is implemented by way of concrete graph element classes
> > >> shown
> > >>> here:
> > >>>
> > >>> https://github.com/ShiftLeftSecurity/tinkergraph-
> > >> gremlin/tree/master/src/test/java/org/apache/tinkerpop/
> > >> gremlin/tinkergraph/structure/specialized/gratefuldead
> > >>>
> > >>> 3. Given that approach, you give up some flexibility in favor of
> > improved
> > >>> memory usage
> > >>> 4. This approach started to feel sufficiently different to me to
> > warrant
> > >>> this improvement as being more than just a "performance enhancement"
> > and
> > >>> more like a fundamental change to what TinkerGraph is about.
> > >>> 5. Of course, Michael had said on the user mailing list that the
> strict
> > >>> schema was optional - though you needed to use it to get the improved
> > >>> memory usage.
> > >>> 6. SoI think the questions forming in my mind are: (a) do we
> want a
> > >>> major new feature (schema support) on TinkerGraph? (b) if so, is the
> > >> schema
> > >>> support implemented in the manner in which we would want it (i.e.
> > >> concrete
> > >>> classes)? (c) is this new feature changing TinkerGraph's mission in
> > >>> TinkerPop? (d) if so, should it simply remain as a fork (presumably
> > >> under a
> > >>> different name to avoid confusion) so that it is not bound by what
> > >>> TinkerGraph is meant to be and can develop more freely?.
> > >>>
> > >>> I'll leave it at that for now and see what other people think.
> > >>>
> > >>> Irrespective of how this ends, I'd quickly offer another round of
> > thanks
> > >> to
> > >>> Michael and his colleagues for coming up with a neat feature and
> > >>> performance enhancement that could be useful to the TinkerPop
> > community.
> > >>>
> > >>
> > >>
> > >
> >
> >
>

Re: [DISCUSS] TinkerGraph Memory Usage Enhancements

[Robert Dale]

I think this 'schema mode' is a good addition.

Robert Dale

On Sun, Jan 14, 2018 at 3:41 AM, Michael Pollmeier <
mich...@michaelpollmeier.com> wrote:

> First of all: if you're not manually enabling the memory-efficient mode,
> you get the exact same behaviour as now.
>
> With the memory-efficient mode enabled, a few semantics change (by design):
> * setting a property that is not schema-conform may (IMO *should*, but
> that depends on the implementation) throw an error
> * generic properties (and meta properties) change conceptually, but you
> can still have them
>
> There's no changes to the underlying queries necessary. We didn't have
> to adapt ours, anyway.
>
> On 13/01/18 11:34, Robert Dale wrote:
> > Michael, are there any limitations other considerations? That is, are
> there
> > any graph features that are lost or schema doesn't apply to, e.g.
> > meta-properties?
> >
> > Robert Dale
> >
> > On Thu, Jan 11, 2018 at 4:03 PM, Michael Pollmeier <
> > mich...@michaelpollmeier.com> wrote:
> >
> >> Thanks Stephen. Just adding a few notes:
> >>
> >> * As mentioned before I'm happy to provide a PR to the Apache
> >> Tinkergraph and thereby close down our fork. If others prefer to not
> >> dilute the waters, I'll equally happily maintain the fork and bring in
> >> changes from Tinkerpop releases (as I've just done with 3.3.1)
> >>
> >> * By default, even our fork still uses the same (generic, non-strict)
> >> mode. The user can enable the specialised/strict/memory-efficient mode
> >> by providing factories for their specialised elements
> >>
> >> * It's all-or-nothing: either all elements are specialised or all
> >> elements are generic. Technically this can be changed, I just didn't
> >> have the need and time to do it.
> >>
> >> Cheers
> >> Michael
> >>
> >>
> >>
> >> On 12/01/18 00:50, Stephen Mallette wrote:
> >>> Michael Pollmeier recently authored a blog post that described how
> their
> >>> fork of TinkerGraph memory usage could be reduced by 70% assuming usage
> >> of
> >>> a strict schema:
> >>>
> >>> https://blog.shiftleft.io/open-sourcing-our-specialized-
> >> tinkergraph-with-70-memory-reduction-and-strict-schema-
> >> validation-fa5cfb3dd82d
> >>>
> >>> The question at this point, is whether or not similar enhancements
> should
> >>> be made to TinkerPop's TinkerGraph. I've had a few minutes to get to
> >>> understand the changes that make the memory improvement possible -
> here's
> >>> my thoughts:
> >>>
> >>> 1. This was a clever way to extend TinkerGraph.
> >>> 2. The schema is implemented by way of concrete graph element classes
> >> shown
> >>> here:
> >>>
> >>> https://github.com/ShiftLeftSecurity/tinkergraph-
> >> gremlin/tree/master/src/test/java/org/apache/tinkerpop/
> >> gremlin/tinkergraph/structure/specialized/gratefuldead
> >>>
> >>> 3. Given that approach, you give up some flexibility in favor of
> improved
> >>> memory usage
> >>> 4. This approach started to feel sufficiently different to me to
> warrant
> >>> this improvement as being more than just a "performance enhancement"
> and
> >>> more like a fundamental change to what TinkerGraph is about.
> >>> 5. Of course, Michael had said on the user mailing list that the strict
> >>> schema was optional - though you needed to use it to get the improved
> >>> memory usage.
> >>> 6. SoI think the questions forming in my mind are: (a) do we want a
> >>> major new feature (schema support) on TinkerGraph? (b) if so, is the
> >> schema
> >>> support implemented in the manner in which we would want it (i.e.
> >> concrete
> >>> classes)? (c) is this new feature changing TinkerGraph's mission in
> >>> TinkerPop? (d) if so, should it simply remain as a fork (presumably
> >> under a
> >>> different name to avoid confusion) so that it is not bound by what
> >>> TinkerGraph is meant to be and can develop more freely?.
> >>>
> >>> I'll leave it at that for now and see what other people think.
> >>>
> >>> Irrespective of how this ends, I'd quickly offer another round of
> thanks
> >> to
> >>> Michael and his colleagues for coming up with a neat feature and
> >>> performance enhancement that could be useful to the TinkerPop
> community.
> >>>
> >>
> >>
> >
>
>

Re: [DISCUSS] TinkerGraph Memory Usage Enhancements

[Michael Pollmeier]

First of all: if you're not manually enabling the memory-efficient mode,
you get the exact same behaviour as now.

With the memory-efficient mode enabled, a few semantics change (by design):
* setting a property that is not schema-conform may (IMO *should*, but
that depends on the implementation) throw an error
* generic properties (and meta properties) change conceptually, but you
can still have them

There's no changes to the underlying queries necessary. We didn't have
to adapt ours, anyway.

On 13/01/18 11:34, Robert Dale wrote:
> Michael, are there any limitations other considerations? That is, are there
> any graph features that are lost or schema doesn't apply to, e.g.
> meta-properties?
> 
> Robert Dale
> 
> On Thu, Jan 11, 2018 at 4:03 PM, Michael Pollmeier <
> mich...@michaelpollmeier.com> wrote:
> 
>> Thanks Stephen. Just adding a few notes:
>>
>> * As mentioned before I'm happy to provide a PR to the Apache
>> Tinkergraph and thereby close down our fork. If others prefer to not
>> dilute the waters, I'll equally happily maintain the fork and bring in
>> changes from Tinkerpop releases (as I've just done with 3.3.1)
>>
>> * By default, even our fork still uses the same (generic, non-strict)
>> mode. The user can enable the specialised/strict/memory-efficient mode
>> by providing factories for their specialised elements
>>
>> * It's all-or-nothing: either all elements are specialised or all
>> elements are generic. Technically this can be changed, I just didn't
>> have the need and time to do it.
>>
>> Cheers
>> Michael
>>
>>
>>
>> On 12/01/18 00:50, Stephen Mallette wrote:
>>> Michael Pollmeier recently authored a blog post that described how their
>>> fork of TinkerGraph memory usage could be reduced by 70% assuming usage
>> of
>>> a strict schema:
>>>
>>> https://blog.shiftleft.io/open-sourcing-our-specialized-
>> tinkergraph-with-70-memory-reduction-and-strict-schema-
>> validation-fa5cfb3dd82d
>>>
>>> The question at this point, is whether or not similar enhancements should
>>> be made to TinkerPop's TinkerGraph. I've had a few minutes to get to
>>> understand the changes that make the memory improvement possible - here's
>>> my thoughts:
>>>
>>> 1. This was a clever way to extend TinkerGraph.
>>> 2. The schema is implemented by way of concrete graph element classes
>> shown
>>> here:
>>>
>>> https://github.com/ShiftLeftSecurity/tinkergraph-
>> gremlin/tree/master/src/test/java/org/apache/tinkerpop/
>> gremlin/tinkergraph/structure/specialized/gratefuldead
>>>
>>> 3. Given that approach, you give up some flexibility in favor of improved
>>> memory usage
>>> 4. This approach started to feel sufficiently different to me to warrant
>>> this improvement as being more than just a "performance enhancement" and
>>> more like a fundamental change to what TinkerGraph is about.
>>> 5. Of course, Michael had said on the user mailing list that the strict
>>> schema was optional - though you needed to use it to get the improved
>>> memory usage.
>>> 6. SoI think the questions forming in my mind are: (a) do we want a
>>> major new feature (schema support) on TinkerGraph? (b) if so, is the
>> schema
>>> support implemented in the manner in which we would want it (i.e.
>> concrete
>>> classes)? (c) is this new feature changing TinkerGraph's mission in
>>> TinkerPop? (d) if so, should it simply remain as a fork (presumably
>> under a
>>> different name to avoid confusion) so that it is not bound by what
>>> TinkerGraph is meant to be and can develop more freely?.
>>>
>>> I'll leave it at that for now and see what other people think.
>>>
>>> Irrespective of how this ends, I'd quickly offer another round of thanks
>> to
>>> Michael and his colleagues for coming up with a neat feature and
>>> performance enhancement that could be useful to the TinkerPop community.
>>>
>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] TinkerGraph Memory Usage Enhancements

[Michael Pollmeier]

Thanks Stephen. Just adding a few notes:

* As mentioned before I'm happy to provide a PR to the Apache
Tinkergraph and thereby close down our fork. If others prefer to not
dilute the waters, I'll equally happily maintain the fork and bring in
changes from Tinkerpop releases (as I've just done with 3.3.1)

* By default, even our fork still uses the same (generic, non-strict)
mode. The user can enable the specialised/strict/memory-efficient mode
by providing factories for their specialised elements

* It's all-or-nothing: either all elements are specialised or all
elements are generic. Technically this can be changed, I just didn't
have the need and time to do it.

Cheers
Michael



On 12/01/18 00:50, Stephen Mallette wrote:
> Michael Pollmeier recently authored a blog post that described how their
> fork of TinkerGraph memory usage could be reduced by 70% assuming usage of
> a strict schema:
> 
> https://blog.shiftleft.io/open-sourcing-our-specialized-tinkergraph-with-70-memory-reduction-and-strict-schema-validation-fa5cfb3dd82d
> 
> The question at this point, is whether or not similar enhancements should
> be made to TinkerPop's TinkerGraph. I've had a few minutes to get to
> understand the changes that make the memory improvement possible - here's
> my thoughts:
> 
> 1. This was a clever way to extend TinkerGraph.
> 2. The schema is implemented by way of concrete graph element classes shown
> here:
> 
> https://github.com/ShiftLeftSecurity/tinkergraph-gremlin/tree/master/src/test/java/org/apache/tinkerpop/gremlin/tinkergraph/structure/specialized/gratefuldead
> 
> 3. Given that approach, you give up some flexibility in favor of improved
> memory usage
> 4. This approach started to feel sufficiently different to me to warrant
> this improvement as being more than just a "performance enhancement" and
> more like a fundamental change to what TinkerGraph is about.
> 5. Of course, Michael had said on the user mailing list that the strict
> schema was optional - though you needed to use it to get the improved
> memory usage.
> 6. SoI think the questions forming in my mind are: (a) do we want a
> major new feature (schema support) on TinkerGraph? (b) if so, is the schema
> support implemented in the manner in which we would want it (i.e. concrete
> classes)? (c) is this new feature changing TinkerGraph's mission in
> TinkerPop? (d) if so, should it simply remain as a fork (presumably under a
> different name to avoid confusion) so that it is not bound by what
> TinkerGraph is meant to be and can develop more freely?.
> 
> I'll leave it at that for now and see what other people think.
> 
> Irrespective of how this ends, I'd quickly offer another round of thanks to
> Michael and his colleagues for coming up with a neat feature and
> performance enhancement that could be useful to the TinkerPop community.
> 



signature.asc
Description: OpenPGP digital signature