Re: [Neo4j] Paged traversal REST API suggestion for improvement
Hi Aseem, Jim, thanks for the explanation. I understand your constraints, but thinking about it more, I'm actually even more baffled -- how can we actually make use of this paged traversal API in a web scenario? Neo4j's traversers (which is what the REST paging API really is) knows nothing about the back button. It is simply a way of getting bite sized chunks of results back to the client while minimising client, server, and network overhead. It's more discrete streaming than a hypermedia set of pages. If your client wants to provide navigation, then it can. But it is your client's job to cache previous pages that it's pulled from the server. The server does not support prev navigation because the aggregate costs to the server are prohibitive. [snip] It seems to me that the root cause of this dilemma is the IterableT constraint you mentioned -- it only goes forward. Yes it does, and for good performance reasons. Perhaps a better API/model then would be to use an offset parameter. This is different than page because it doesn't require the server to keep state; it just tells the server, return me the results starting *after* this result (which would generally be the last result I saw on my previous page of results). The offset would require us to go forward through the iterator to the offset. Then return from there. I'd considered it briefly, but then dropped the idea since it's extremely wasteful. - Get me the results of this traverse would return everything That's what the normal (non-paged) traversal API does. - Get me the results of this traverse, paged with size 20 would return 20 results That's what the paged traversal API does. - Get me the results of this traverse, paged with size 20, starting with the node/rel after this one [given by an ID] would return the next 20 results That's what (sort of) repeated GETs on the traversal API does, with the caveat that you already specified the page size when you set up the traverser. But we can't give you the ability to look up an arbitrary page on the server. That doesn't even make sense since between invocations the graph may mutate and you'll get different results back anyway. Jim PS - if you want to be prev friendly, then think about making the entire prev page cacheable rather than placing another call on the server for the previous page and doing the hard work of rendering it all over again. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Paged traversal REST API suggestion for improvement
All of this works, without JS. That means the client (your browser) isn't the party doing the paging or keeping track of all results. It's something on the server side. That something on the server side is called Google. That is, there is an organisation which has set up databases, processing, caching, and a myriad of other things and exposed them through a Web server. Saying Neo4j should do this because Google does isn't a fair comparison - Google has more than a database, it has an entire application. I'd say your app should do this, and if you can find a performant way of doing it that doesn't hold server-side state then I'll be happy to take a patch for the REST API. Jim ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Paged traversal REST API suggestion for improvement
Hi Aseem, When you GET a resource, you're asking for its state at a particular instant in time. GET's don't always return the same content, what's important about them is that clients can't be held responsible for any (unintended) side-effects of a GET operation. For example, if you GET the BBC new page now, and then GET it again in 20 minutes, the bytes you receive will almost certainly be different. I think what's perhaps a bit odd in the paged traverser case is that clients do in fact trigger a state transition on GET which materially changes the response. Adding /next to the paged traverser URI does not change this, it just lengthens the URI. Under the covers, the REST API is just using the traversal API and so gets back an IterableT which is forward-only. We can't (or rather won't) do things like ?page=2 (or similar) because that would involve holding all the traversal state on the server which is one of the things paged traversers were meant to avoid (being kind to both the server and clients). If you want to remember all traversal state on the client, that's OK. You can page through your own local data structures. In fact given the number of clients is likely to be larger than the number of servers, this makes sense since the memory burden is shared amongst the many clients, rather than being placed on the single server. So, I'll buy that we've conflated a little too much into a single GET, but it's motivated by a sincere trade off to not overburden the server. Jim ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Paged traversal REST API suggestion for improvement
Jim, thanks for the explanation. I understand your constraints, but thinking about it more, I'm actually even more baffled -- how can we actually make use of this paged traversal API in a web scenario? [neo4j db] [web server] [web client, e.g. browser] If I want to show the results of a search, let's say, in a way that supports paging, it just seems like I can't take advantage of the Neo4j API at all here. If the user wants to be able to refresh the page, or click Prev, the browser isn't keeping state across page changes, so it has to rely on the web server to give it the right page. If the server now has to keep state of all results, it defeats the purpose of using the Neo4j paging API altogether: the server necessarily has to fetch everything and handle paging itself. Am I missing something? Or is this (typical in our case) scenario not something you guys are designing for? (And if not, what *is* the use case you guys envisioned for this?) It seems to me that the root cause of this dilemma is the IterableT constraint you mentioned -- it only goes forward. Perhaps a better API/model then would be to use an offset parameter. This is different than page because it doesn't require the server to keep state; it just tells the server, return me the results starting *after* this result (which would generally be the last result I saw on my previous page of results). E.g. the following requests: - Get me the results of this traverse would return everything - Get me the results of this traverse, paged with size 20 would return 20 results - Get me the results of this traverse, paged with size 20, starting with the node/rel after this one [given by an ID] would return the next 20 results - etc. Aseem On Thu, Jul 28, 2011 at 5:09 AM, Jim Webber j...@neotechnology.com wrote: Hi Aseem, When you GET a resource, you're asking for its state at a particular instant in time. GET's don't always return the same content, what's important about them is that clients can't be held responsible for any (unintended) side-effects of a GET operation. For example, if you GET the BBC new page now, and then GET it again in 20 minutes, the bytes you receive will almost certainly be different. I think what's perhaps a bit odd in the paged traverser case is that clients do in fact trigger a state transition on GET which materially changes the response. Adding /next to the paged traverser URI does not change this, it just lengthens the URI. Under the covers, the REST API is just using the traversal API and so gets back an IterableT which is forward-only. We can't (or rather won't) do things like ?page=2 (or similar) because that would involve holding all the traversal state on the server which is one of the things paged traversers were meant to avoid (being kind to both the server and clients). If you want to remember all traversal state on the client, that's OK. You can page through your own local data structures. In fact given the number of clients is likely to be larger than the number of servers, this makes sense since the memory burden is shared amongst the many clients, rather than being placed on the single server. So, I'll buy that we've conflated a little too much into a single GET, but it's motivated by a sincere trade off to not overburden the server. Jim ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Paged traversal REST API suggestion for improvement
If you don't keep state paging will not work properly if the data changes often. What may have been record #21 when you are viewing the first page of 20 result might not be record #21 when you go to fetch the 2nd page. If you're only concerned with pages of 20 or so, you should absolutely, positively do the paging on the browser. Anything up to a couple thousand rows can *easily* be handled by any modern browser. Keep the parsed JSON data on the client side and page it there. This approach will be very performant on both the client and server, and won't require any state. -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Aseem Kishore Sent: Thursday, July 28, 2011 3:01 PM To: Neo4j user discussions Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement Jim, thanks for the explanation. I understand your constraints, but thinking about it more, I'm actually even more baffled -- how can we actually make use of this paged traversal API in a web scenario? [neo4j db] [web server] [web client, e.g. browser] If I want to show the results of a search, let's say, in a way that supports paging, it just seems like I can't take advantage of the Neo4j API at all here. If the user wants to be able to refresh the page, or click Prev, the browser isn't keeping state across page changes, so it has to rely on the web server to give it the right page. If the server now has to keep state of all results, it defeats the purpose of using the Neo4j paging API altogether: the server necessarily has to fetch everything and handle paging itself. Am I missing something? Or is this (typical in our case) scenario not something you guys are designing for? (And if not, what *is* the use case you guys envisioned for this?) It seems to me that the root cause of this dilemma is the IterableT constraint you mentioned -- it only goes forward. Perhaps a better API/model then would be to use an offset parameter. This is different than page because it doesn't require the server to keep state; it just tells the server, return me the results starting *after* this result (which would generally be the last result I saw on my previous page of results). E.g. the following requests: - Get me the results of this traverse would return everything - Get me the results of this traverse, paged with size 20 would return 20 results - Get me the results of this traverse, paged with size 20, starting with the node/rel after this one [given by an ID] would return the next 20 results - etc. Aseem On Thu, Jul 28, 2011 at 5:09 AM, Jim Webber j...@neotechnology.com wrote: Hi Aseem, When you GET a resource, you're asking for its state at a particular instant in time. GET's don't always return the same content, what's important about them is that clients can't be held responsible for any (unintended) side-effects of a GET operation. For example, if you GET the BBC new page now, and then GET it again in 20 minutes, the bytes you receive will almost certainly be different. I think what's perhaps a bit odd in the paged traverser case is that clients do in fact trigger a state transition on GET which materially changes the response. Adding /next to the paged traverser URI does not change this, it just lengthens the URI. Under the covers, the REST API is just using the traversal API and so gets back an IterableT which is forward-only. We can't (or rather won't) do things like ?page=2 (or similar) because that would involve holding all the traversal state on the server which is one of the things paged traversers were meant to avoid (being kind to both the server and clients). If you want to remember all traversal state on the client, that's OK. You can page through your own local data structures. In fact given the number of clients is likely to be larger than the number of servers, this makes sense since the memory burden is shared amongst the many clients, rather than being placed on the single server. So, I'll buy that we've conflated a little too much into a single GET, but it's motivated by a sincere trade off to not overburden the server. Jim ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Paged traversal REST API suggestion for improvement
BTW, paging is a relic of the dial-up modem days, IMNSHO. If a machine is the client of the REST API call, it should get all the data in a single, atomic call. If it is a browser or mobile app, there's a natural limit to the # of items that a human will bother paging through, which is fairly small (20 - 500 typically), so again, it is painless to get all the data in a single call. -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Rick Bullotta Sent: Thursday, July 28, 2011 6:11 PM To: Neo4j user discussions Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement If you don't keep state paging will not work properly if the data changes often. What may have been record #21 when you are viewing the first page of 20 result might not be record #21 when you go to fetch the 2nd page. If you're only concerned with pages of 20 or so, you should absolutely, positively do the paging on the browser. Anything up to a couple thousand rows can *easily* be handled by any modern browser. Keep the parsed JSON data on the client side and page it there. This approach will be very performant on both the client and server, and won't require any state. -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Aseem Kishore Sent: Thursday, July 28, 2011 3:01 PM To: Neo4j user discussions Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement Jim, thanks for the explanation. I understand your constraints, but thinking about it more, I'm actually even more baffled -- how can we actually make use of this paged traversal API in a web scenario? [neo4j db] [web server] [web client, e.g. browser] If I want to show the results of a search, let's say, in a way that supports paging, it just seems like I can't take advantage of the Neo4j API at all here. If the user wants to be able to refresh the page, or click Prev, the browser isn't keeping state across page changes, so it has to rely on the web server to give it the right page. If the server now has to keep state of all results, it defeats the purpose of using the Neo4j paging API altogether: the server necessarily has to fetch everything and handle paging itself. Am I missing something? Or is this (typical in our case) scenario not something you guys are designing for? (And if not, what *is* the use case you guys envisioned for this?) It seems to me that the root cause of this dilemma is the IterableT constraint you mentioned -- it only goes forward. Perhaps a better API/model then would be to use an offset parameter. This is different than page because it doesn't require the server to keep state; it just tells the server, return me the results starting *after* this result (which would generally be the last result I saw on my previous page of results). E.g. the following requests: - Get me the results of this traverse would return everything - Get me the results of this traverse, paged with size 20 would return 20 results - Get me the results of this traverse, paged with size 20, starting with the node/rel after this one [given by an ID] would return the next 20 results - etc. Aseem On Thu, Jul 28, 2011 at 5:09 AM, Jim Webber j...@neotechnology.com wrote: Hi Aseem, When you GET a resource, you're asking for its state at a particular instant in time. GET's don't always return the same content, what's important about them is that clients can't be held responsible for any (unintended) side-effects of a GET operation. For example, if you GET the BBC new page now, and then GET it again in 20 minutes, the bytes you receive will almost certainly be different. I think what's perhaps a bit odd in the paged traverser case is that clients do in fact trigger a state transition on GET which materially changes the response. Adding /next to the paged traverser URI does not change this, it just lengthens the URI. Under the covers, the REST API is just using the traversal API and so gets back an IterableT which is forward-only. We can't (or rather won't) do things like ?page=2 (or similar) because that would involve holding all the traversal state on the server which is one of the things paged traversers were meant to avoid (being kind to both the server and clients). If you want to remember all traversal state on the client, that's OK. You can page through your own local data structures. In fact given the number of clients is likely to be larger than the number of servers, this makes sense since the memory burden is shared amongst the many clients, rather than being placed on the single server. So, I'll buy that we've conflated a little too much into a single GET, but it's motivated by a sincere trade off to not overburden the server. Jim ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo
Re: [Neo4j] Paged traversal REST API suggestion for improvement
Sure but isn’t it a huge waste of bandwidth if you load hundreds of results and the user only looks at the first dozen? On Thu, Jul 28, 2011 at 15:16, Rick Bullotta rick.bullo...@thingworx.comwrote: BTW, paging is a relic of the dial-up modem days, IMNSHO. If a machine is the client of the REST API call, it should get all the data in a single, atomic call. If it is a browser or mobile app, there's a natural limit to the # of items that a human will bother paging through, which is fairly small (20 - 500 typically), so again, it is painless to get all the data in a single call. -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Rick Bullotta Sent: Thursday, July 28, 2011 6:11 PM To: Neo4j user discussions Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement If you don't keep state paging will not work properly if the data changes often. What may have been record #21 when you are viewing the first page of 20 result might not be record #21 when you go to fetch the 2nd page. If you're only concerned with pages of 20 or so, you should absolutely, positively do the paging on the browser. Anything up to a couple thousand rows can *easily* be handled by any modern browser. Keep the parsed JSON data on the client side and page it there. This approach will be very performant on both the client and server, and won't require any state. -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Aseem Kishore Sent: Thursday, July 28, 2011 3:01 PM To: Neo4j user discussions Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement Jim, thanks for the explanation. I understand your constraints, but thinking about it more, I'm actually even more baffled -- how can we actually make use of this paged traversal API in a web scenario? [neo4j db] [web server] [web client, e.g. browser] If I want to show the results of a search, let's say, in a way that supports paging, it just seems like I can't take advantage of the Neo4j API at all here. If the user wants to be able to refresh the page, or click Prev, the browser isn't keeping state across page changes, so it has to rely on the web server to give it the right page. If the server now has to keep state of all results, it defeats the purpose of using the Neo4j paging API altogether: the server necessarily has to fetch everything and handle paging itself. Am I missing something? Or is this (typical in our case) scenario not something you guys are designing for? (And if not, what *is* the use case you guys envisioned for this?) It seems to me that the root cause of this dilemma is the IterableT constraint you mentioned -- it only goes forward. Perhaps a better API/model then would be to use an offset parameter. This is different than page because it doesn't require the server to keep state; it just tells the server, return me the results starting *after* this result (which would generally be the last result I saw on my previous page of results). E.g. the following requests: - Get me the results of this traverse would return everything - Get me the results of this traverse, paged with size 20 would return 20 results - Get me the results of this traverse, paged with size 20, starting with the node/rel after this one [given by an ID] would return the next 20 results - etc. Aseem On Thu, Jul 28, 2011 at 5:09 AM, Jim Webber j...@neotechnology.com wrote: Hi Aseem, When you GET a resource, you're asking for its state at a particular instant in time. GET's don't always return the same content, what's important about them is that clients can't be held responsible for any (unintended) side-effects of a GET operation. For example, if you GET the BBC new page now, and then GET it again in 20 minutes, the bytes you receive will almost certainly be different. I think what's perhaps a bit odd in the paged traverser case is that clients do in fact trigger a state transition on GET which materially changes the response. Adding /next to the paged traverser URI does not change this, it just lengthens the URI. Under the covers, the REST API is just using the traversal API and so gets back an IterableT which is forward-only. We can't (or rather won't) do things like ?page=2 (or similar) because that would involve holding all the traversal state on the server which is one of the things paged traversers were meant to avoid (being kind to both the server and clients). If you want to remember all traversal state on the client, that's OK. You can page through your own local data structures. In fact given the number of clients is likely to be larger than the number of servers, this makes sense since the memory burden is shared amongst the many clients, rather than being placed on the single server. So
Re: [Neo4j] Paged traversal REST API suggestion for improvement
Not really, unless the JSON content is HUGE - which is rare. The small price in bandwidth versus a responsive UI and the need for not managing state is a good tradeoff. The only exception might be mobile apps, but in that case, I'd just fetch a smaller block (maybe 4 or 5 pages worth). Very few users have the patience to page through more than a few pages worth of information. -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Daniel Gasienica Sent: Thursday, July 28, 2011 6:20 PM To: Neo4j user discussions Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement Sure but isn’t it a huge waste of bandwidth if you load hundreds of results and the user only looks at the first dozen? On Thu, Jul 28, 2011 at 15:16, Rick Bullotta rick.bullo...@thingworx.comwrote: BTW, paging is a relic of the dial-up modem days, IMNSHO. If a machine is the client of the REST API call, it should get all the data in a single, atomic call. If it is a browser or mobile app, there's a natural limit to the # of items that a human will bother paging through, which is fairly small (20 - 500 typically), so again, it is painless to get all the data in a single call. -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Rick Bullotta Sent: Thursday, July 28, 2011 6:11 PM To: Neo4j user discussions Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement If you don't keep state paging will not work properly if the data changes often. What may have been record #21 when you are viewing the first page of 20 result might not be record #21 when you go to fetch the 2nd page. If you're only concerned with pages of 20 or so, you should absolutely, positively do the paging on the browser. Anything up to a couple thousand rows can *easily* be handled by any modern browser. Keep the parsed JSON data on the client side and page it there. This approach will be very performant on both the client and server, and won't require any state. -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Aseem Kishore Sent: Thursday, July 28, 2011 3:01 PM To: Neo4j user discussions Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement Jim, thanks for the explanation. I understand your constraints, but thinking about it more, I'm actually even more baffled -- how can we actually make use of this paged traversal API in a web scenario? [neo4j db] [web server] [web client, e.g. browser] If I want to show the results of a search, let's say, in a way that supports paging, it just seems like I can't take advantage of the Neo4j API at all here. If the user wants to be able to refresh the page, or click Prev, the browser isn't keeping state across page changes, so it has to rely on the web server to give it the right page. If the server now has to keep state of all results, it defeats the purpose of using the Neo4j paging API altogether: the server necessarily has to fetch everything and handle paging itself. Am I missing something? Or is this (typical in our case) scenario not something you guys are designing for? (And if not, what *is* the use case you guys envisioned for this?) It seems to me that the root cause of this dilemma is the IterableT constraint you mentioned -- it only goes forward. Perhaps a better API/model then would be to use an offset parameter. This is different than page because it doesn't require the server to keep state; it just tells the server, return me the results starting *after* this result (which would generally be the last result I saw on my previous page of results). E.g. the following requests: - Get me the results of this traverse would return everything - Get me the results of this traverse, paged with size 20 would return 20 results - Get me the results of this traverse, paged with size 20, starting with the node/rel after this one [given by an ID] would return the next 20 results - etc. Aseem On Thu, Jul 28, 2011 at 5:09 AM, Jim Webber j...@neotechnology.com wrote: Hi Aseem, When you GET a resource, you're asking for its state at a particular instant in time. GET's don't always return the same content, what's important about them is that clients can't be held responsible for any (unintended) side-effects of a GET operation. For example, if you GET the BBC new page now, and then GET it again in 20 minutes, the bytes you receive will almost certainly be different. I think what's perhaps a bit odd in the paged traverser case is that clients do in fact trigger a state transition on GET which materially changes the response. Adding /next to the paged traverser URI does not change this, it just lengthens the URI. Under the covers, the REST API is just using the traversal API
Re: [Neo4j] Paged traversal REST API suggestion for improvement
Sorry, I don't understand. Are you going to tell Google and Bing to send JSON for their search results also, and page on the client side? There is a very valid use case for not wanting to require JavaScript to page. Aseem On Thu, Jul 28, 2011 at 3:25 PM, Rick Bullotta rick.bullo...@thingworx.comwrote: Not really, unless the JSON content is HUGE - which is rare. The small price in bandwidth versus a responsive UI and the need for not managing state is a good tradeoff. The only exception might be mobile apps, but in that case, I'd just fetch a smaller block (maybe 4 or 5 pages worth). Very few users have the patience to page through more than a few pages worth of information. -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Daniel Gasienica Sent: Thursday, July 28, 2011 6:20 PM To: Neo4j user discussions Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement Sure but isn’t it a huge waste of bandwidth if you load hundreds of results and the user only looks at the first dozen? On Thu, Jul 28, 2011 at 15:16, Rick Bullotta rick.bullo...@thingworx.com wrote: BTW, paging is a relic of the dial-up modem days, IMNSHO. If a machine is the client of the REST API call, it should get all the data in a single, atomic call. If it is a browser or mobile app, there's a natural limit to the # of items that a human will bother paging through, which is fairly small (20 - 500 typically), so again, it is painless to get all the data in a single call. -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Rick Bullotta Sent: Thursday, July 28, 2011 6:11 PM To: Neo4j user discussions Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement If you don't keep state paging will not work properly if the data changes often. What may have been record #21 when you are viewing the first page of 20 result might not be record #21 when you go to fetch the 2nd page. If you're only concerned with pages of 20 or so, you should absolutely, positively do the paging on the browser. Anything up to a couple thousand rows can *easily* be handled by any modern browser. Keep the parsed JSON data on the client side and page it there. This approach will be very performant on both the client and server, and won't require any state. -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Aseem Kishore Sent: Thursday, July 28, 2011 3:01 PM To: Neo4j user discussions Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement Jim, thanks for the explanation. I understand your constraints, but thinking about it more, I'm actually even more baffled -- how can we actually make use of this paged traversal API in a web scenario? [neo4j db] [web server] [web client, e.g. browser] If I want to show the results of a search, let's say, in a way that supports paging, it just seems like I can't take advantage of the Neo4j API at all here. If the user wants to be able to refresh the page, or click Prev, the browser isn't keeping state across page changes, so it has to rely on the web server to give it the right page. If the server now has to keep state of all results, it defeats the purpose of using the Neo4j paging API altogether: the server necessarily has to fetch everything and handle paging itself. Am I missing something? Or is this (typical in our case) scenario not something you guys are designing for? (And if not, what *is* the use case you guys envisioned for this?) It seems to me that the root cause of this dilemma is the IterableT constraint you mentioned -- it only goes forward. Perhaps a better API/model then would be to use an offset parameter. This is different than page because it doesn't require the server to keep state; it just tells the server, return me the results starting *after* this result (which would generally be the last result I saw on my previous page of results). E.g. the following requests: - Get me the results of this traverse would return everything - Get me the results of this traverse, paged with size 20 would return 20 results - Get me the results of this traverse, paged with size 20, starting with the node/rel after this one [given by an ID] would return the next 20 results - etc. Aseem On Thu, Jul 28, 2011 at 5:09 AM, Jim Webber j...@neotechnology.com wrote: Hi Aseem, When you GET a resource, you're asking for its state at a particular instant in time. GET's don't always return the same content, what's important about them is that clients can't be held responsible for any (unintended) side-effects of a GET operation. For example, if you GET the BBC new page now, and then GET
Re: [Neo4j] Paged traversal REST API suggestion for improvement
Um, I'm guessing that you aren't aware of how Google's UI/API works... Open up Firebug, Chrome Tools, or Fiddler, and you'll see that the biggest chunk of traffic when you switch from one page of Google search results to another page (via the main Google search page) is a JSON packet...which uses their search API...which returns JSON. Do they page? Yes. Can you call the API and get hundreds of results at once? Yes... -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Aseem Kishore Sent: Thursday, July 28, 2011 6:42 PM To: Neo4j user discussions Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement Sorry, I don't understand. Are you going to tell Google and Bing to send JSON for their search results also, and page on the client side? There is a very valid use case for not wanting to require JavaScript to page. Aseem On Thu, Jul 28, 2011 at 3:25 PM, Rick Bullotta rick.bullo...@thingworx.comwrote: Not really, unless the JSON content is HUGE - which is rare. The small price in bandwidth versus a responsive UI and the need for not managing state is a good tradeoff. The only exception might be mobile apps, but in that case, I'd just fetch a smaller block (maybe 4 or 5 pages worth). Very few users have the patience to page through more than a few pages worth of information. -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Daniel Gasienica Sent: Thursday, July 28, 2011 6:20 PM To: Neo4j user discussions Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement Sure but isn't it a huge waste of bandwidth if you load hundreds of results and the user only looks at the first dozen? On Thu, Jul 28, 2011 at 15:16, Rick Bullotta rick.bullo...@thingworx.com wrote: BTW, paging is a relic of the dial-up modem days, IMNSHO. If a machine is the client of the REST API call, it should get all the data in a single, atomic call. If it is a browser or mobile app, there's a natural limit to the # of items that a human will bother paging through, which is fairly small (20 - 500 typically), so again, it is painless to get all the data in a single call. -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Rick Bullotta Sent: Thursday, July 28, 2011 6:11 PM To: Neo4j user discussions Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement If you don't keep state paging will not work properly if the data changes often. What may have been record #21 when you are viewing the first page of 20 result might not be record #21 when you go to fetch the 2nd page. If you're only concerned with pages of 20 or so, you should absolutely, positively do the paging on the browser. Anything up to a couple thousand rows can *easily* be handled by any modern browser. Keep the parsed JSON data on the client side and page it there. This approach will be very performant on both the client and server, and won't require any state. -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Aseem Kishore Sent: Thursday, July 28, 2011 3:01 PM To: Neo4j user discussions Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement Jim, thanks for the explanation. I understand your constraints, but thinking about it more, I'm actually even more baffled -- how can we actually make use of this paged traversal API in a web scenario? [neo4j db] [web server] [web client, e.g. browser] If I want to show the results of a search, let's say, in a way that supports paging, it just seems like I can't take advantage of the Neo4j API at all here. If the user wants to be able to refresh the page, or click Prev, the browser isn't keeping state across page changes, so it has to rely on the web server to give it the right page. If the server now has to keep state of all results, it defeats the purpose of using the Neo4j paging API altogether: the server necessarily has to fetch everything and handle paging itself. Am I missing something? Or is this (typical in our case) scenario not something you guys are designing for? (And if not, what *is* the use case you guys envisioned for this?) It seems to me that the root cause of this dilemma is the IterableT constraint you mentioned -- it only goes forward. Perhaps a better API/model then would be to use an offset parameter. This is different than page because it doesn't require the server to keep state; it just tells the server, return me the results starting *after* this result (which would generally be the last result I saw on my previous page of results). E.g. the following requests: - Get me the results of this traverse would return
Re: [Neo4j] Paged traversal REST API suggestion for improvement
Haha. I think we're speaking past each other. Here's my point: Disable JS. Open up Google. Do a search. Click to the next page. Click to back to the previous page. Click page 3. Refresh. Etc. All of this works, without JS. That means the client (your browser) isn't the party doing the paging or keeping track of all results. It's something on the server side. My original question was: I don't see how one can achieve this using Neo4j's paged traversal API. Is that not a reasonable scenario / use case for it? If not, what is? My follow-up suggestion was: could we not support this scenario / use case by changing (or expanding) the API to support an offset parameter? In other words, random access -- like keying into a graph by index and then following edges -- instead of forward-only and once-only IterableT. Aseem On Thu, Jul 28, 2011 at 4:27 PM, Rick Bullotta rick.bullo...@thingworx.comwrote: Um, I'm guessing that you aren't aware of how Google's UI/API works... Open up Firebug, Chrome Tools, or Fiddler, and you'll see that the biggest chunk of traffic when you switch from one page of Google search results to another page (via the main Google search page) is a JSON packet...which uses their search API...which returns JSON. Do they page? Yes. Can you call the API and get hundreds of results at once? Yes... -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Aseem Kishore Sent: Thursday, July 28, 2011 6:42 PM To: Neo4j user discussions Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement Sorry, I don't understand. Are you going to tell Google and Bing to send JSON for their search results also, and page on the client side? There is a very valid use case for not wanting to require JavaScript to page. Aseem On Thu, Jul 28, 2011 at 3:25 PM, Rick Bullotta rick.bullo...@thingworx.comwrote: Not really, unless the JSON content is HUGE - which is rare. The small price in bandwidth versus a responsive UI and the need for not managing state is a good tradeoff. The only exception might be mobile apps, but in that case, I'd just fetch a smaller block (maybe 4 or 5 pages worth). Very few users have the patience to page through more than a few pages worth of information. -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On Behalf Of Daniel Gasienica Sent: Thursday, July 28, 2011 6:20 PM To: Neo4j user discussions Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement Sure but isn't it a huge waste of bandwidth if you load hundreds of results and the user only looks at the first dozen? On Thu, Jul 28, 2011 at 15:16, Rick Bullotta rick.bullo...@thingworx.com wrote: BTW, paging is a relic of the dial-up modem days, IMNSHO. If a machine is the client of the REST API call, it should get all the data in a single, atomic call. If it is a browser or mobile app, there's a natural limit to the # of items that a human will bother paging through, which is fairly small (20 - 500 typically), so again, it is painless to get all the data in a single call. -Original Message- From: user-boun...@lists.neo4j.org [mailto: user-boun...@lists.neo4j.org] On Behalf Of Rick Bullotta Sent: Thursday, July 28, 2011 6:11 PM To: Neo4j user discussions Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement If you don't keep state paging will not work properly if the data changes often. What may have been record #21 when you are viewing the first page of 20 result might not be record #21 when you go to fetch the 2nd page. If you're only concerned with pages of 20 or so, you should absolutely, positively do the paging on the browser. Anything up to a couple thousand rows can *easily* be handled by any modern browser. Keep the parsed JSON data on the client side and page it there. This approach will be very performant on both the client and server, and won't require any state. -Original Message- From: user-boun...@lists.neo4j.org [mailto: user-boun...@lists.neo4j.org] On Behalf Of Aseem Kishore Sent: Thursday, July 28, 2011 3:01 PM To: Neo4j user discussions Subject: Re: [Neo4j] Paged traversal REST API suggestion for improvement Jim, thanks for the explanation. I understand your constraints, but thinking about it more, I'm actually even more baffled -- how can we actually make use of this paged traversal API in a web scenario? [neo4j db] [web server] [web client, e.g. browser] If I want to show the results of a search, let's say, in a way that supports paging, it just seems like I can't take advantage of the Neo4j API at all here. If the user wants to be able to refresh the page, or click Prev, the browser isn't keeping
