Re: [Neo4j] Dealing with time when writes from many clients
actually, ignore that scenario and problem setup. i'm being daft. my comment on "rich" client API still valid though. it's my one pain-point at the moment. On Thu, Jul 14, 2011 at 11:43 AM, Patrik Sundberg wrote: > Hi, > > Scenario: > I want to save a new version of a node. Versions are organized by time, and > consists of a node connected to a reference node via a relationship with a > valid range. I enforce the rule that you can't change the past. So I will > need to take the current time, check vs existing version relationships (and > fail if the time is not > the start time of each existing version), and if > we're ok create the new node and attach to the reference node via a new > relationship. > > Problem: > When I say "the current time" above it's not well defined how that is > determined. In the scenario where many clients work vs a server then the > options are either: > 1. clients ask a time server for the current time > 2. assume clients have synchronized time keeping outside of application > > For case 1 we'd need a transaction that I don't think is possible to > implement via REST api (in same transaction ask for time, depending on time > and data in DB make decisions, then write new data to db). Case 2 seems a > bit flaky. > > Any suggestions for how to tackle something like this? > > I think I'm again running up against cases where not having a client API > with transactions makes life difficult for developers. Scanning the > mailinglist I can see a lot of people using the REST api. I doubt they are > super tied to REST, probably more likely to a client-server setup. Seems > reasonable to assume they are also likely to run up against cases where not > having "normal" transactions in client API will be problematic. I think a > rich client API incl transactions (and using a fast protocol) would push a > lot more projects towards neo4j. Any chance of seeing something like that in > the near future? 1.5? > > Thanks for a great product, with a few tweaks like this I think it'll be > even better and gain more and more users. > > Patrik > > ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] Dealing with time when writes from many clients
Hi, Scenario: I want to save a new version of a node. Versions are organized by time, and consists of a node connected to a reference node via a relationship with a valid range. I enforce the rule that you can't change the past. So I will need to take the current time, check vs existing version relationships (and fail if the time is not > the start time of each existing version), and if we're ok create the new node and attach to the reference node via a new relationship. Problem: When I say "the current time" above it's not well defined how that is determined. In the scenario where many clients work vs a server then the options are either: 1. clients ask a time server for the current time 2. assume clients have synchronized time keeping outside of application For case 1 we'd need a transaction that I don't think is possible to implement via REST api (in same transaction ask for time, depending on time and data in DB make decisions, then write new data to db). Case 2 seems a bit flaky. Any suggestions for how to tackle something like this? I think I'm again running up against cases where not having a client API with transactions makes life difficult for developers. Scanning the mailinglist I can see a lot of people using the REST api. I doubt they are super tied to REST, probably more likely to a client-server setup. Seems reasonable to assume they are also likely to run up against cases where not having "normal" transactions in client API will be problematic. I think a rich client API incl transactions (and using a fast protocol) would push a lot more projects towards neo4j. Any chance of seeing something like that in the near future? 1.5? Thanks for a great product, with a few tweaks like this I think it'll be even better and gain more and more users. Patrik ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] clean database / unit tests
Nope, but I can get it for you if you give me some pointers where it should
occur (just started to play with neo4j so not yet up to speed where
everything goes and when).
On Sun, Jul 10, 2011 at 5:30 PM, Michael Hunger <
michael.hun...@neotechnology.com> wrote:
> thanks so much will look into it
>
> do you by chance have the server stacktrace that produced the internal
> server error (500)
>
> Michael
>
> mobile mail please excuse brevity and typos
>
> Am 10.07.2011 um 17:13 schrieb Patrik Sundberg >:
>
> > I just removed the call to JSONObject.toString(result) and replaced it
> with
> > "{}" and it now behaves. So something's going wrong with the JSON
> > conversion.
> >
> > I'm not familiar enough to dig much deeper, but hopefully that'll let
> > someone else figure out how to fix it. In the meantime returning empty
> JSON
> > result is fine for my purposes so I'll carry on with my tests.
> >
> > On Sun, Jul 10, 2011 at 3:51 PM, Patrik Sundberg
> > wrote:
> >
> >> I just redid the same exercise with 1.4M06. No issues whatsoever with
> >> compiling the extension, but still getting code 500 responses.
> >>
> >> The tests for the plugin itself runs ok. So seems it runs fine in the
> >> localserver that gets compiled with the plugin but not when I drop it in
> >> with the 1.4M06 server. Some sort of dependency issue?
> >>
> >> I just updated the pom.xml to be 1.4M06 and changed nothing else when
> >> compiling it.
> >>
> >>
> >>
> >> On Sat, Jul 9, 2011 at 7:10 PM, Patrik Sundberg <
> patrik.sundb...@gmail.com
> >>> wrote:
> >>
> >>> I tried building it for 1.4M05 and after some fiddling around with
> maven
> >>> it seemed to build alright. Copied jar and enabled in the config.
> Restarted
> >>> server, and seems to load fine.
> >>>
> >>> Trying to use it I get the message "WARNING: Deleted DatabaseL " as
> >>> expected, but then I get back code 500 Internal Server Error.
> >>>
> >>> Looking at the code that seems to indicate that this line throws an
> >>> exception:
> >>>
> >>>
> https://github.com/jexp/neo4j-clean-remote-db-addon/blob/master/src/main/java/org/neo4j/server/extension/test/delete/DeleteDatabaseResource.java#L64
> >>>
> >>> Any idea what's going on there?
> >>>
> >>> Another peculiar thing: the webadmin interface suggests it did not
> delete
> >>> things in the DB, rather it created more nodes, properties and
> >>> relationships. Highly confusing.. Showing 47 nodes, 126 properties and
> 31
> >>> relationships when there were never that many things there before the
> >>> /cleandb call.
> >>>
> >>> Subsequent calls to /cleandb outputs that nothing is deleted (still
> >>> returning code 500 though). Leaves me thinking it's the webadmin that
> is
> >>> confused rather than that it didn't clean the db.
> >>>
> >>> Left confused.
> >>>
> >>>
> >>>
> >>> On Fri, Jul 8, 2011 at 9:19 PM, Michael Hunger <
> >>> michael.hun...@neotechnology.com> wrote:
> >>>
> >>>> Even then I don't think that left over relationship types do any
> damage
> >>>> in testing.
> >>>>
> >>>> The addon has a way to delete the database completely but that's only
> >>>> used for large db's (>10k nodes). I could make it accessible
> >>>> programmatically.
> >>>> But I don't think it's worth the effort.
> >>>>
> >>>> Cheers
> >>>>
> >>>> Michael
> >>>>
> >>>> Am 08.07.2011 um 22:13 schrieb Mattias Persson:
> >>>>
> >>>>> Relationship types are never deleted from a neo4j database. That's
> >>>> because
> >>>>> it doesn't track (globally) how many relationships there are of any
> >>>> given
> >>>>> type, so deleting a type when there are no more such relationships
> left
> >>>>> isn't possible and pretty much only useful in cases like these, for
> >>>> testing.
> >>>>>
> >>>>> 2011/7/8 Boris Kizelshteyn
> >>>>>
> >>>>>> I just started using this myself and it's pretty great exce
Re: [Neo4j] clean database / unit tests
I just removed the call to JSONObject.toString(result) and replaced it with
"{}" and it now behaves. So something's going wrong with the JSON
conversion.
I'm not familiar enough to dig much deeper, but hopefully that'll let
someone else figure out how to fix it. In the meantime returning empty JSON
result is fine for my purposes so I'll carry on with my tests.
On Sun, Jul 10, 2011 at 3:51 PM, Patrik Sundberg
wrote:
> I just redid the same exercise with 1.4M06. No issues whatsoever with
> compiling the extension, but still getting code 500 responses.
>
> The tests for the plugin itself runs ok. So seems it runs fine in the
> localserver that gets compiled with the plugin but not when I drop it in
> with the 1.4M06 server. Some sort of dependency issue?
>
> I just updated the pom.xml to be 1.4M06 and changed nothing else when
> compiling it.
>
>
>
> On Sat, Jul 9, 2011 at 7:10 PM, Patrik Sundberg > wrote:
>
>> I tried building it for 1.4M05 and after some fiddling around with maven
>> it seemed to build alright. Copied jar and enabled in the config. Restarted
>> server, and seems to load fine.
>>
>> Trying to use it I get the message "WARNING: Deleted DatabaseL " as
>> expected, but then I get back code 500 Internal Server Error.
>>
>> Looking at the code that seems to indicate that this line throws an
>> exception:
>>
>> https://github.com/jexp/neo4j-clean-remote-db-addon/blob/master/src/main/java/org/neo4j/server/extension/test/delete/DeleteDatabaseResource.java#L64
>>
>> Any idea what's going on there?
>>
>> Another peculiar thing: the webadmin interface suggests it did not delete
>> things in the DB, rather it created more nodes, properties and
>> relationships. Highly confusing.. Showing 47 nodes, 126 properties and 31
>> relationships when there were never that many things there before the
>> /cleandb call.
>>
>> Subsequent calls to /cleandb outputs that nothing is deleted (still
>> returning code 500 though). Leaves me thinking it's the webadmin that is
>> confused rather than that it didn't clean the db.
>>
>> Left confused.
>>
>>
>>
>> On Fri, Jul 8, 2011 at 9:19 PM, Michael Hunger <
>> michael.hun...@neotechnology.com> wrote:
>>
>>> Even then I don't think that left over relationship types do any damage
>>> in testing.
>>>
>>> The addon has a way to delete the database completely but that's only
>>> used for large db's (>10k nodes). I could make it accessible
>>> programmatically.
>>> But I don't think it's worth the effort.
>>>
>>> Cheers
>>>
>>> Michael
>>>
>>> Am 08.07.2011 um 22:13 schrieb Mattias Persson:
>>>
>>> > Relationship types are never deleted from a neo4j database. That's
>>> because
>>> > it doesn't track (globally) how many relationships there are of any
>>> given
>>> > type, so deleting a type when there are no more such relationships left
>>> > isn't possible and pretty much only useful in cases like these, for
>>> testing.
>>> >
>>> > 2011/7/8 Boris Kizelshteyn
>>> >
>>> >> I just started using this myself and it's pretty great except that it
>>> seems
>>> >> to leave behind the "relationship types":
>>> >>
>>> >> curl -v GET http://localhost:7474/db/data/relationship/types
>>> >>
>>> >> I'm on 1.3, any thoughts?
>>> >>
>>> >> Thanks!
>>> >>
>>> >> On Fri, Jul 8, 2011 at 11:56 AM, Patrik Sundberg
>>> >> wrote:
>>> >>
>>> >>> great, just what i was looking for.
>>> >>>
>>> >>> thanks
>>> >>>
>>> >>>
>>> >>> On Fri, Jul 8, 2011 at 3:37 PM, Jim Webber
>>> >> wrote:
>>> >>>
>>> >>>> Hi Patrik,
>>> >>>>
>>> >>>> Michael Hunger's add-on is what you need:
>>> >>>>
>>> >>>> https://github.com/jexp/neo4j-clean-remote-db-addon
>>> >>>>
>>> >>>> I believe it'll be packaged by default with our next release (but
>>> >> totally
>>> >>>> disabled!).
>>> >>>>
>>> >>>> Jim
>>> >>>>
>&
Re: [Neo4j] clean database / unit tests
I just redid the same exercise with 1.4M06. No issues whatsoever with compiling the extension, but still getting code 500 responses. The tests for the plugin itself runs ok. So seems it runs fine in the localserver that gets compiled with the plugin but not when I drop it in with the 1.4M06 server. Some sort of dependency issue? I just updated the pom.xml to be 1.4M06 and changed nothing else when compiling it. On Sat, Jul 9, 2011 at 7:10 PM, Patrik Sundberg wrote: > I tried building it for 1.4M05 and after some fiddling around with maven it > seemed to build alright. Copied jar and enabled in the config. Restarted > server, and seems to load fine. > > Trying to use it I get the message "WARNING: Deleted DatabaseL " as > expected, but then I get back code 500 Internal Server Error. > > Looking at the code that seems to indicate that this line throws an > exception: > > https://github.com/jexp/neo4j-clean-remote-db-addon/blob/master/src/main/java/org/neo4j/server/extension/test/delete/DeleteDatabaseResource.java#L64 > > Any idea what's going on there? > > Another peculiar thing: the webadmin interface suggests it did not delete > things in the DB, rather it created more nodes, properties and > relationships. Highly confusing.. Showing 47 nodes, 126 properties and 31 > relationships when there were never that many things there before the > /cleandb call. > > Subsequent calls to /cleandb outputs that nothing is deleted (still > returning code 500 though). Leaves me thinking it's the webadmin that is > confused rather than that it didn't clean the db. > > Left confused. > > > > On Fri, Jul 8, 2011 at 9:19 PM, Michael Hunger < > michael.hun...@neotechnology.com> wrote: > >> Even then I don't think that left over relationship types do any damage in >> testing. >> >> The addon has a way to delete the database completely but that's only used >> for large db's (>10k nodes). I could make it accessible programmatically. >> But I don't think it's worth the effort. >> >> Cheers >> >> Michael >> >> Am 08.07.2011 um 22:13 schrieb Mattias Persson: >> >> > Relationship types are never deleted from a neo4j database. That's >> because >> > it doesn't track (globally) how many relationships there are of any >> given >> > type, so deleting a type when there are no more such relationships left >> > isn't possible and pretty much only useful in cases like these, for >> testing. >> > >> > 2011/7/8 Boris Kizelshteyn >> > >> >> I just started using this myself and it's pretty great except that it >> seems >> >> to leave behind the "relationship types": >> >> >> >> curl -v GET http://localhost:7474/db/data/relationship/types >> >> >> >> I'm on 1.3, any thoughts? >> >> >> >> Thanks! >> >> >> >> On Fri, Jul 8, 2011 at 11:56 AM, Patrik Sundberg >> >> wrote: >> >> >> >>> great, just what i was looking for. >> >>> >> >>> thanks >> >>> >> >>> >> >>> On Fri, Jul 8, 2011 at 3:37 PM, Jim Webber >> >> wrote: >> >>> >> >>>> Hi Patrik, >> >>>> >> >>>> Michael Hunger's add-on is what you need: >> >>>> >> >>>> https://github.com/jexp/neo4j-clean-remote-db-addon >> >>>> >> >>>> I believe it'll be packaged by default with our next release (but >> >> totally >> >>>> disabled!). >> >>>> >> >>>> Jim >> >>>> >> >>>> On 8 Jul 2011, at 15:32, Patrik Sundberg wrote: >> >>>> >> >>>>> Hi, >> >>>>> >> >>>>> Is there a good way to ensure I have a clean database? I'm thinking >> >> in >> >>>> the >> >>>>> context of running some unit tests, I'd like to have some >> >>> setup/teardown >> >>>>> hooks that ensures a clean database for various tests. >> >>>>> >> >>>>> I'm thinking for a client using the REST API here. I can see how I >> >>> could >> >>>>> write a plugin that is exposed via REST that deletes all nodes and >> >>> rels, >> >>>> was >> >>>>> just wondering if someone has already do
Re: [Neo4j] clean database / unit tests
I tried building it for 1.4M05 and after some fiddling around with maven it seemed to build alright. Copied jar and enabled in the config. Restarted server, and seems to load fine. Trying to use it I get the message "WARNING: Deleted DatabaseL " as expected, but then I get back code 500 Internal Server Error. Looking at the code that seems to indicate that this line throws an exception: https://github.com/jexp/neo4j-clean-remote-db-addon/blob/master/src/main/java/org/neo4j/server/extension/test/delete/DeleteDatabaseResource.java#L64 Any idea what's going on there? Another peculiar thing: the webadmin interface suggests it did not delete things in the DB, rather it created more nodes, properties and relationships. Highly confusing.. Showing 47 nodes, 126 properties and 31 relationships when there were never that many things there before the /cleandb call. Subsequent calls to /cleandb outputs that nothing is deleted (still returning code 500 though). Leaves me thinking it's the webadmin that is confused rather than that it didn't clean the db. Left confused. On Fri, Jul 8, 2011 at 9:19 PM, Michael Hunger < michael.hun...@neotechnology.com> wrote: > Even then I don't think that left over relationship types do any damage in > testing. > > The addon has a way to delete the database completely but that's only used > for large db's (>10k nodes). I could make it accessible programmatically. > But I don't think it's worth the effort. > > Cheers > > Michael > > Am 08.07.2011 um 22:13 schrieb Mattias Persson: > > > Relationship types are never deleted from a neo4j database. That's > because > > it doesn't track (globally) how many relationships there are of any given > > type, so deleting a type when there are no more such relationships left > > isn't possible and pretty much only useful in cases like these, for > testing. > > > > 2011/7/8 Boris Kizelshteyn > > > >> I just started using this myself and it's pretty great except that it > seems > >> to leave behind the "relationship types": > >> > >> curl -v GET http://localhost:7474/db/data/relationship/types > >> > >> I'm on 1.3, any thoughts? > >> > >> Thanks! > >> > >> On Fri, Jul 8, 2011 at 11:56 AM, Patrik Sundberg > >> wrote: > >> > >>> great, just what i was looking for. > >>> > >>> thanks > >>> > >>> > >>> On Fri, Jul 8, 2011 at 3:37 PM, Jim Webber > >> wrote: > >>> > >>>> Hi Patrik, > >>>> > >>>> Michael Hunger's add-on is what you need: > >>>> > >>>> https://github.com/jexp/neo4j-clean-remote-db-addon > >>>> > >>>> I believe it'll be packaged by default with our next release (but > >> totally > >>>> disabled!). > >>>> > >>>> Jim > >>>> > >>>> On 8 Jul 2011, at 15:32, Patrik Sundberg wrote: > >>>> > >>>>> Hi, > >>>>> > >>>>> Is there a good way to ensure I have a clean database? I'm thinking > >> in > >>>> the > >>>>> context of running some unit tests, I'd like to have some > >>> setup/teardown > >>>>> hooks that ensures a clean database for various tests. > >>>>> > >>>>> I'm thinking for a client using the REST API here. I can see how I > >>> could > >>>>> write a plugin that is exposed via REST that deletes all nodes and > >>> rels, > >>>> was > >>>>> just wondering if someone has already done this? > >>>>> > >>>>> Thanks, > >>>>> Patrik > >>>>> ___ > >>>>> Neo4j mailing list > >>>>> User@lists.neo4j.org > >>>>> https://lists.neo4j.org/mailman/listinfo/user > >>>> > >>>> ___ > >>>> Neo4j mailing list > >>>> User@lists.neo4j.org > >>>> https://lists.neo4j.org/mailman/listinfo/user > >>>> > >>> ___ > >>> Neo4j mailing list > >>> User@lists.neo4j.org > >>> https://lists.neo4j.org/mailman/listinfo/user > >>> > >> ___ > >> Neo4j mailing list > >> User@lists.neo4j.org > >> https://lists.neo4j.org/mailman/listinfo/user > >> > > > > > > > > -- > > Mattias Persson, [matt...@neotechnology.com] > > Hacker, Neo Technology > > www.neotechnology.com > > ___ > > Neo4j mailing list > > User@lists.neo4j.org > > https://lists.neo4j.org/mailman/listinfo/user > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] clean database / unit tests
great, just what i was looking for. thanks On Fri, Jul 8, 2011 at 3:37 PM, Jim Webber wrote: > Hi Patrik, > > Michael Hunger's add-on is what you need: > > https://github.com/jexp/neo4j-clean-remote-db-addon > > I believe it'll be packaged by default with our next release (but totally > disabled!). > > Jim > > On 8 Jul 2011, at 15:32, Patrik Sundberg wrote: > > > Hi, > > > > Is there a good way to ensure I have a clean database? I'm thinking in > the > > context of running some unit tests, I'd like to have some setup/teardown > > hooks that ensures a clean database for various tests. > > > > I'm thinking for a client using the REST API here. I can see how I could > > write a plugin that is exposed via REST that deletes all nodes and rels, > was > > just wondering if someone has already done this? > > > > Thanks, > > Patrik > > ___ > > Neo4j mailing list > > User@lists.neo4j.org > > https://lists.neo4j.org/mailman/listinfo/user > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] clean database / unit tests
Hi, Is there a good way to ensure I have a clean database? I'm thinking in the context of running some unit tests, I'd like to have some setup/teardown hooks that ensures a clean database for various tests. I'm thinking for a client using the REST API here. I can see how I could write a plugin that is exposed via REST that deletes all nodes and rels, was just wondering if someone has already done this? Thanks, Patrik ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] REST batch support - transaction support for java rest client?
yes, i'll see if i get time to go down that path. it's mostly to keep the programmer happy and productive, not a 100% requirement. On Thu, Jul 7, 2011 at 9:58 PM, Michael Hunger < michael.hun...@neotechnology.com> wrote: > Right those are some of the issues. > > So one way would be to specify a tx timeout upfront which automatically > rolls back the tx (you can just add a kind of timer/TTL to the tx-session > object) and clears it as well. > > Keeping state on the server is always a problem but I don't see a different > solution for that. But it might worth a try especially if it helps you with > your concrete scenario. > > Michael > > > > Am 07.07.2011 um 15:30 schrieb Patrik Sundberg: > > > good idea. i'll ponder it for a bit. > > > > but yes, we clearly need to keep state around, so for REST it'd be > carried > > around in session. but on server side I guess you have issues with never > > ending transactions, how to cull them, etc. since it's a stateless > > req/response comm channel. on a permanent channel it's easy to detect > > disconnect and clean up, over http not as easy. > > > > thanks > > > > > > On Thu, Jul 7, 2011 at 12:42 PM, Michael Hunger < > > michael.hun...@neotechnology.com> wrote: > > > >> But then it would be possible to write a RequestFilter for the > Neo4j-Server > >> that does start and commit/rollbacks transactions. > >> > >> I.e. you create a tx object and put it in the session-context if there > is > >> none and return a tx-token that the filter uses (e.g. as header-field). > >> then later you can pull it out again and attach it to the current thread > >> (that's the tricky part). > >> On commit or rollback you just do that with the tx (after attaching it > to > >> the thread). > >> > >> As the RestfulGraphDb and the Filter share the same execution thread > this > >> could/should work. > >> > >> I wouldn't want to support that in the neo4j server by default as this > >> creates a lot of server-side state that has to be managed. > >> > >> But if it works out one could publish that as server-extension. > >> > >> HTH > >> > >> Michael > >> > >> Am 07.07.2011 um 13:30 schrieb Patrik Sundberg: > >> > >>> Following up on the topic of transactions for client API. > >>> > >>> What is the current plan for some sort of client side API supporting > >>> transactions? > >>> > >>> I'm playing around with some ideas here and the lack of transaction > >> support > >>> in the client API is problematic. I know there's BATCH support in the > >> REST > >>> API which effectively is a transaction, but it doesn't always suit. For > >>> example I have the following steps that I'd like to accomplish: > >>> - create a reference node > >>> - check if a node with a given domain id exist in an index, if it does, > >> fail > >>> - create an entity node for the given domain id > >>> - add entity node to the index > >>> - attach entity node to ref node > >>> - create a node representing a specific version of the entity node > >>> - attach the version node to the entity node, with some properties on > the > >>> relationships signifying valid time > >>> > >>> That should all be considered an atomic operation, all or nothing. > Doing > >> it > >>> step by step is very easy and natural with REST API, but trying to roll > >> back > >>> on error is flaky. > >>> > >>> I think could batch it, but from a programming style it becomes pretty > >>> unnatural. Same thing with a plugin for doing the steps. The natural > flow > >> of > >>> code client side gets distorted by having to collect a lot of data > >> upfront > >>> and then provide all that data to a method call. It's doable, just > >> doesn't > >>> seem ideal. > >>> > >>> Using an embedded db, exposing as some sort of service etc is also > >> doable, > >>> it's just that my domain is graph related and I'm pretty happy with > just > >> the > >>> "primitives" and using a remote server (if I could have transactions). > >>> Number of clients are quite a few and need to share their data + don't > >> all > &
Re: [Neo4j] REST batch support - transaction support for java rest client?
good idea. i'll ponder it for a bit. but yes, we clearly need to keep state around, so for REST it'd be carried around in session. but on server side I guess you have issues with never ending transactions, how to cull them, etc. since it's a stateless req/response comm channel. on a permanent channel it's easy to detect disconnect and clean up, over http not as easy. thanks On Thu, Jul 7, 2011 at 12:42 PM, Michael Hunger < michael.hun...@neotechnology.com> wrote: > But then it would be possible to write a RequestFilter for the Neo4j-Server > that does start and commit/rollbacks transactions. > > I.e. you create a tx object and put it in the session-context if there is > none and return a tx-token that the filter uses (e.g. as header-field). > then later you can pull it out again and attach it to the current thread > (that's the tricky part). > On commit or rollback you just do that with the tx (after attaching it to > the thread). > > As the RestfulGraphDb and the Filter share the same execution thread this > could/should work. > > I wouldn't want to support that in the neo4j server by default as this > creates a lot of server-side state that has to be managed. > > But if it works out one could publish that as server-extension. > > HTH > > Michael > > Am 07.07.2011 um 13:30 schrieb Patrik Sundberg: > > > Following up on the topic of transactions for client API. > > > > What is the current plan for some sort of client side API supporting > > transactions? > > > > I'm playing around with some ideas here and the lack of transaction > support > > in the client API is problematic. I know there's BATCH support in the > REST > > API which effectively is a transaction, but it doesn't always suit. For > > example I have the following steps that I'd like to accomplish: > > - create a reference node > > - check if a node with a given domain id exist in an index, if it does, > fail > > - create an entity node for the given domain id > > - add entity node to the index > > - attach entity node to ref node > > - create a node representing a specific version of the entity node > > - attach the version node to the entity node, with some properties on the > > relationships signifying valid time > > > > That should all be considered an atomic operation, all or nothing. Doing > it > > step by step is very easy and natural with REST API, but trying to roll > back > > on error is flaky. > > > > I think could batch it, but from a programming style it becomes pretty > > unnatural. Same thing with a plugin for doing the steps. The natural flow > of > > code client side gets distorted by having to collect a lot of data > upfront > > and then provide all that data to a method call. It's doable, just > doesn't > > seem ideal. > > > > Using an embedded db, exposing as some sort of service etc is also > doable, > > it's just that my domain is graph related and I'm pretty happy with just > the > > "primitives" and using a remote server (if I could have transactions). > > Number of clients are quite a few and need to share their data + don't > all > > run all the time so can't make the client API the embedded api. > > > > I'd think it's not an uncommon situation and many people wishing for a > > support for natural client side transaction API (similar to embedded > api). > > > > Patrik > > > > > > On Tue, Jul 5, 2011 at 12:27 PM, Patrik Sundberg > > wrote: > > > >> yeah, harder problem than my first hunch. > >> > >> sounds like plugins is the way to go for now, hopefully introduction of > >> non-rest protocol with same interface as embedded API in 1.5 will > simplify > >> things in the future. > >> > >> thanks > >> > >> > >> On Mon, Jul 4, 2011 at 11:07 PM, Michael Hunger < > >> michael.hun...@neotechnology.com> wrote: > >> > >>> Patrick, > >>> > >>> I've already thought long and hard about that. > >>> > >>> The problem is you can't implement that transparently as you can never > >>> allow code in a second call rely on data derived from a previous one. > >>> > >>> The simplest form that I came up with is a "BatchCommand" that gets an > API > >>> interface injected that allows requests but doesn't return data. > >>> > >>> The execution of this Batch command would then return a "BatchResult
Re: [Neo4j] REST batch support - transaction support for java rest client?
Following up on the topic of transactions for client API. What is the current plan for some sort of client side API supporting transactions? I'm playing around with some ideas here and the lack of transaction support in the client API is problematic. I know there's BATCH support in the REST API which effectively is a transaction, but it doesn't always suit. For example I have the following steps that I'd like to accomplish: - create a reference node - check if a node with a given domain id exist in an index, if it does, fail - create an entity node for the given domain id - add entity node to the index - attach entity node to ref node - create a node representing a specific version of the entity node - attach the version node to the entity node, with some properties on the relationships signifying valid time That should all be considered an atomic operation, all or nothing. Doing it step by step is very easy and natural with REST API, but trying to roll back on error is flaky. I think could batch it, but from a programming style it becomes pretty unnatural. Same thing with a plugin for doing the steps. The natural flow of code client side gets distorted by having to collect a lot of data upfront and then provide all that data to a method call. It's doable, just doesn't seem ideal. Using an embedded db, exposing as some sort of service etc is also doable, it's just that my domain is graph related and I'm pretty happy with just the "primitives" and using a remote server (if I could have transactions). Number of clients are quite a few and need to share their data + don't all run all the time so can't make the client API the embedded api. I'd think it's not an uncommon situation and many people wishing for a support for natural client side transaction API (similar to embedded api). Patrik On Tue, Jul 5, 2011 at 12:27 PM, Patrik Sundberg wrote: > yeah, harder problem than my first hunch. > > sounds like plugins is the way to go for now, hopefully introduction of > non-rest protocol with same interface as embedded API in 1.5 will simplify > things in the future. > > thanks > > > On Mon, Jul 4, 2011 at 11:07 PM, Michael Hunger < > michael.hun...@neotechnology.com> wrote: > >> Patrick, >> >> I've already thought long and hard about that. >> >> The problem is you can't implement that transparently as you can never >> allow code in a second call rely on data derived from a previous one. >> >> The simplest form that I came up with is a "BatchCommand" that gets an API >> interface injected that allows requests but doesn't return data. >> >> The execution of this Batch command would then return a "BatchResult" with >> all the data acquired during the batch operation. >> >> Another way would be to inject the normal GraphDatabaseService interface, >> record the invocations in a first phase and then execute the batch command >> again (this time ignoring the inputs but then returning the results) but >> this is bad from a usability perspective. >> >> One critical issue is the creation of relationships as they depend on the >> correct node-ids of previously created nodes. Jacob already thought about >> some means of referring to previous output data but I think kept away from >> that as we didn't want to make this batch-interface a turing complete >> language. >> >> So you see, it's not that simple. >> >> Michael >> >> Am 27.06.2011 um 20:45 schrieb Patrik Sundberg: >> >> > Hi, >> > >> > Since there is now possible to send off batches of operations via the >> REST >> > interface, I was wondering if anyone has started to look at implementing >> > transactions in the java REST client ( >> > https://github.com/jexp/neo4j-java-rest-binding) ? >> > >> > It would seem possible, but I can also see it could involve some major >> > reorganizing of the internals of the client to make everything aware of >> > transactions and submit via batch command. >> > >> > Patrik >> > ___ >> > Neo4j mailing list >> > User@lists.neo4j.org >> > https://lists.neo4j.org/mailman/listinfo/user >> >> ___ >> Neo4j mailing list >> User@lists.neo4j.org >> https://lists.neo4j.org/mailman/listinfo/user >> > > ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] REST batch support - transaction support for java rest client?
yeah, harder problem than my first hunch. sounds like plugins is the way to go for now, hopefully introduction of non-rest protocol with same interface as embedded API in 1.5 will simplify things in the future. thanks On Mon, Jul 4, 2011 at 11:07 PM, Michael Hunger < michael.hun...@neotechnology.com> wrote: > Patrick, > > I've already thought long and hard about that. > > The problem is you can't implement that transparently as you can never > allow code in a second call rely on data derived from a previous one. > > The simplest form that I came up with is a "BatchCommand" that gets an API > interface injected that allows requests but doesn't return data. > > The execution of this Batch command would then return a "BatchResult" with > all the data acquired during the batch operation. > > Another way would be to inject the normal GraphDatabaseService interface, > record the invocations in a first phase and then execute the batch command > again (this time ignoring the inputs but then returning the results) but > this is bad from a usability perspective. > > One critical issue is the creation of relationships as they depend on the > correct node-ids of previously created nodes. Jacob already thought about > some means of referring to previous output data but I think kept away from > that as we didn't want to make this batch-interface a turing complete > language. > > So you see, it's not that simple. > > Michael > > Am 27.06.2011 um 20:45 schrieb Patrik Sundberg: > > > Hi, > > > > Since there is now possible to send off batches of operations via the > REST > > interface, I was wondering if anyone has started to look at implementing > > transactions in the java REST client ( > > https://github.com/jexp/neo4j-java-rest-binding) ? > > > > It would seem possible, but I can also see it could involve some major > > reorganizing of the internals of the client to make everything aware of > > transactions and submit via batch command. > > > > Patrik > > ___ > > Neo4j mailing list > > User@lists.neo4j.org > > https://lists.neo4j.org/mailman/listinfo/user > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] REST batch support - transaction support for java rest client?
Hi, Since there is now possible to send off batches of operations via the REST interface, I was wondering if anyone has started to look at implementing transactions in the java REST client ( https://github.com/jexp/neo4j-java-rest-binding) ? It would seem possible, but I can also see it could involve some major reorganizing of the internals of the client to make everything aware of transactions and submit via batch command. Patrik ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Archiving in time / being able to quickly refer to a historical state
On Wed, Jun 15, 2011 at 2:31 AM, Patrik Sundberg wrote: > On 14 Jun 2011, at 22:51, Michael Hunger > wrote: > > > Patrick, > > > > David Montag created a versioned graph approach on github: > https://github.com/dmontag/neo4j-versioning > > > > Perhaps you can take some inspiration from that. > > I'll take a look, thanks. > I had a look at this. It's a really cool idea. It's in principal a very good solution to what I'd like to accomplish, even more ambitious even. And for part of my graph it's ideal in practical terms as well - the parts that don't change so often and where number of relationships wont mushroom. However, I have a nagging suspicion that it may not be practical in the Book, Elements, Effects use case - may get too many relationships. If I have versioned relationships that keeps growing from Books to Elements then over time a Book will get a lot of direct relationships it's involved in. Let's say I have ~500 elements/book (at any point in time), and they'll change every day (not quite true but lets assume so for now), then a book will add on 125,000 relationships/year (assuming 250 business days). Over a few years you'll start having half a million direct relationships of a single Book. That's probably not an ideal deisgn, correct? What I was thinking is that I could have an ElementSet reference node layer sitting between a Book and Elements. An ElementSet has a timestamp property for when the snapshot was made or the relationship from Book to ElementSet has a valid range on it. Then an ElementSet node in turn has Elements. So in a year you'd add ~250 Book to ElementSet relationships, and each ElementSet would have ~500 relationships to it's Elements. I can still get the state of the book quite easily via a simple 2 level traversal. Another thought I'd be happy to get some input on: My Elements are essentially consisting of an Entity and a Quantity. The Effects as well. If we assume we'll have millions of Effects and 100s of thousands of Elements then I'm thinking that it's not a good idea to have actual relationships pointing to the Entity nodes from the Element and Effect nodes, correct? The Entity nodes would get millions of direct relationships. I could instead put in the identifier or the Entity in the Element and the Effect as properties and use index based queries for that step. Patrik > Am 14.06.2011 um 21:14 schrieb Patrik Sundberg: > >> I should probably put in some estimates of the sort of orders of magnitude >> of data involved: >> Books are in the 100s >> Effects are in the 10,000s / day >> Elements are in the 100s / Book >> >> On Tue, Jun 14, 2011 at 8:07 PM, Patrik Sundberg >> wrote: >> >>> Hi, >>> >>> Design question: >>> I have a Book. A book has Elements. To change what Elements are in a Book >>> there are Effects. An Effect adds or deletes an element from a Book (and has >>> other info). Everyday I archive the state of a Book so that I can easily get >>> the state of the Book (it's Elements) at a historical point in time. The >>> number of elements and which elements are in a Book vary from day to day >>> depending on the Effects taking place in the time period. >>> >>> So it's a bit like "Book contents archive of day X + 1 = Book contents >>> archive of day X + the Effects between X and X + 1" >>> >>> There's no need to represent the daily archives in any way, it's the >>> functionality of having historical snapshots that matter to me, so that one >>> can easily get the state of a Book at any point in time by finding the >>> closest archive point before the time and applying the Effects from that >>> archive timestamp and the point of interest. >>> >>> Ideally want to keep a long time history of Book state and Effects. >>> >>> Any ideas for an effecient of organizing this in a graph friendly way? >>> >>> Thanks, >>> Patrik >>> >>> >> ___ >> Neo4j mailing list >> User@lists.neo4j.org >> https://lists.neo4j.org/mailman/listinfo/user > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Archiving in time / being able to quickly refer to a historical state
On 14 Jun 2011, at 22:51, Michael Hunger wrote: > Patrick, > > David Montag created a versioned graph approach on github: > https://github.com/dmontag/neo4j-versioning > > Perhaps you can take some inspiration from that. I'll take a look, thanks. > Otherwise: > Books and Elements are nodes, effects would be relationships which have a > timestamp (probably indexed). > > The relationship between book - [:CONTAINS]-> element would have a validity > time-span. (It is started and open-ended when the element is added and closed > when the element is removed. > > The you could traverse any book along its CONTAINS relationships while taking > the time-span property into account for a concrete timestamp. > > You might choose to add the effect information as additional relationships > between books and elements but perhaps it is enough to add those to > appropriate properties of the CONTAINS relationship. > I'll have millions of effects but thousands of elements for a book so seems a bit inefficient. Wouldn't something like an intermediate layer of nodes of ElementSet (timestamped) make sense instead? A book has many element sets and an element set has many elements. Effects dictate how element sets evolve over time. > Are there other effects as well? > Is it possible to change the content of an element? (that would complicate > the matter) > Otherwise if that is realized as remove the old element and add the new one > this would suffice. > > Addtionally it could be interesting to have links between related elements > (either structural links - part -> chapter -> section -> paragraph | image | > figure | table) or historical, i.e. link elements that replace other to those. > Nope, no other effects and effects can't change (would be remove and readd to amend). My domain is not documents but trading. So my books are a logical organisation of security holdings, my elements securities and my effects trades. Archiving holdings a way of being more efficient of finding the holding at a given point in time without having to process too many trades. P > HTH > > Michael > > Am 14.06.2011 um 21:14 schrieb Patrik Sundberg: > >> I should probably put in some estimates of the sort of orders of magnitude >> of data involved: >> Books are in the 100s >> Effects are in the 10,000s / day >> Elements are in the 100s / Book >> >> On Tue, Jun 14, 2011 at 8:07 PM, Patrik Sundberg >> wrote: >> >>> Hi, >>> >>> Design question: >>> I have a Book. A book has Elements. To change what Elements are in a Book >>> there are Effects. An Effect adds or deletes an element from a Book (and has >>> other info). Everyday I archive the state of a Book so that I can easily get >>> the state of the Book (it's Elements) at a historical point in time. The >>> number of elements and which elements are in a Book vary from day to day >>> depending on the Effects taking place in the time period. >>> >>> So it's a bit like "Book contents archive of day X + 1 = Book contents >>> archive of day X + the Effects between X and X + 1" >>> >>> There's no need to represent the daily archives in any way, it's the >>> functionality of having historical snapshots that matter to me, so that one >>> can easily get the state of a Book at any point in time by finding the >>> closest archive point before the time and applying the Effects from that >>> archive timestamp and the point of interest. >>> >>> Ideally want to keep a long time history of Book state and Effects. >>> >>> Any ideas for an effecient of organizing this in a graph friendly way? >>> >>> Thanks, >>> Patrik >>> >>> >> ___ >> Neo4j mailing list >> User@lists.neo4j.org >> https://lists.neo4j.org/mailman/listinfo/user > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Archiving in time / being able to quickly refer to a historical state
Yes, the archive is a roll up of effects. And it's mostly for efficiency, so that one does not have to go through all the millions of effects to get the state of a given point in time but can start from a known state and just add 10s of thousands of effects between archive times. On 14 Jun 2011, at 21:28, McKinley wrote: > Can you describe what you mean by archiving the state of the book each day? > It seems that you have a transactional system with your effects and an > archive at a point in time just seems like the range of effects by date. Is > an archive a roll-up of the effects for efficiency or backup? > > Cheers, > > McKinley > > On Tue, Jun 14, 2011 at 12:14 PM, Patrik Sundberg > wrote: >> >>> Design question: >>> I have a Book. A book has Elements. To change what Elements are in a Book >>> there are Effects. An Effect adds or deletes an element from a Book (and >> has >>> other info). Everyday I archive the state of a Book so that I can easily >> get >>> the state of the Book (it's Elements) at a historical point in time. The >>> number of elements and which elements are in a Book vary from day to day >>> depending on the Effects taking place in the time period. >>> >>> So it's a bit like "Book contents archive of day X + 1 = Book contents >>> archive of day X + the Effects between X and X + 1" >>> >>> There's no need to represent the daily archives in any way, it's the >>> functionality of having historical snapshots that matter to me, so that >> one >>> can easily get the state of a Book at any point in time by finding the >>> closest archive point before the time and applying the Effects from that >>> archive timestamp and the point of interest. >> > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Archiving in time / being able to quickly refer to a historical state
I should probably put in some estimates of the sort of orders of magnitude of data involved: Books are in the 100s Effects are in the 10,000s / day Elements are in the 100s / Book On Tue, Jun 14, 2011 at 8:07 PM, Patrik Sundberg wrote: > Hi, > > Design question: > I have a Book. A book has Elements. To change what Elements are in a Book > there are Effects. An Effect adds or deletes an element from a Book (and has > other info). Everyday I archive the state of a Book so that I can easily get > the state of the Book (it's Elements) at a historical point in time. The > number of elements and which elements are in a Book vary from day to day > depending on the Effects taking place in the time period. > > So it's a bit like "Book contents archive of day X + 1 = Book contents > archive of day X + the Effects between X and X + 1" > > There's no need to represent the daily archives in any way, it's the > functionality of having historical snapshots that matter to me, so that one > can easily get the state of a Book at any point in time by finding the > closest archive point before the time and applying the Effects from that > archive timestamp and the point of interest. > > Ideally want to keep a long time history of Book state and Effects. > > Any ideas for an effecient of organizing this in a graph friendly way? > > Thanks, > Patrik > > ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] Archiving in time / being able to quickly refer to a historical state
Hi, Design question: I have a Book. A book has Elements. To change what Elements are in a Book there are Effects. An Effect adds or deletes an element from a Book (and has other info). Everyday I archive the state of a Book so that I can easily get the state of the Book (it's Elements) at a historical point in time. The number of elements and which elements are in a Book vary from day to day depending on the Effects taking place in the time period. So it's a bit like "Book contents archive of day X + 1 = Book contents archive of day X + the Effects between X and X + 1" There's no need to represent the daily archives in any way, it's the functionality of having historical snapshots that matter to me, so that one can easily get the state of a Book at any point in time by finding the closest archive point before the time and applying the Effects from that archive timestamp and the point of interest. Ideally want to keep a long time history of Book state and Effects. Any ideas for an effecient of organizing this in a graph friendly way? Thanks, Patrik ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
