[Neo4j] java heep size in neo4j when importing huge data
hi all, I am trying to import csv data to a new graph database using neo4j.PY after a 65000 row of insertion I am getting a java heep size error knowing that I extended the jvm memory to 640MB. how can I save the data after each 5 row of insertion? how can I use the begintransaction function in python version of neo4j.py? I am using the following script: graphdb = neo4j.GraphDatabase(lparam_database_path, classpath=lparam_kernel_class_path, jvm=lparam_jvm_path, heap_size=ls_jvm_heap_size) with graphdb.transaction: sectornode = Subreference.Node.SECTOR_ROOT(graphdb, label=Sectors, level=1) categorynode = graphdb.node(label=Categories, level=2) productnode = graphdb.node(label=Products, level=3) brandnode = graphdb.node(label=Brands, level=4) subbrandnode = graphdb.node(label=Subbrands, level=5) relationship1 = sectornode.CATEGORY_ROOT(categorynode) relationship2 = categorynode.PRODUCT_ROOT(productnode) relationship3 = productnode.BRAND_ROOT(brandnode) relationship4 = brandnode.SUBBRAND_ROOT(subbrandnode) sectorlabelindex = graphdb.index(sector_label_index, create=True) categorylabelindex = graphdb.index(category_label_index, create=True) productlabelindex = graphdb.index(product_label_index, create=True) brandlabelindex = graphdb.index(brand_label_index, create=True) subbrandlabelindex = graphdb.index(subbrand_label_index, create=True) li_index = 0 with graphdb.transaction: for data in data_list: data[0] = unicode(data[0], 'utf-8') data[1] = unicode(data[1], 'utf-8') data[2] = unicode(data[2], 'utf-8') data[3] = unicode(data[3], 'utf-8') data[4] = unicode(data[4], 'utf-8') li_index = li_index + 1 li_counter = li_counter + 1 ls_message = print importing row + str(li_index) + of + str(li_total_rows) tempsectornode = sectorlabelindex[data[0] + ***level01***] tempcategorynode = categorylabelindex[data[1] + ***level02***] tempproductnode = productlabelindex[data[2] + ***level03***] tempbrandnode = brandlabelindex[data[3] + ***level04***] tempsubbrandnode = subbrandlabelindex[data[4] + ***level05***] try: ls_message = ls_message + tempsectornode[label] + , except: tempsectornode = graphdb.node(label=data[0], level=***level01***) sectorlabelindex[data[0] + ***level01***] = tempsectornode try: ls_message = ls_message + tempcategorynode[label] + , except: tempcategorynode = graphdb.node(label=data[1], level=***level02***) categorylabelindex[data[1] + ***level02***] = tempcategorynode try: ls_message = ls_message + tempproductnode[label] + , except: tempproductnode = graphdb.node(label=data[2], level=***level03***) productlabelindex[data[2] + ***level03***] = tempproductnode try: ls_message = ls_message + tempbrandnode[label] + , except: tempbrandnode = graphdb.node(label=data[3], level=***level04***) brandlabelindex[data[3] + ***level04***] = tempbrandnode try: ls_message = ls_message + tempsubbrandnode[label] + , except: tempsubbrandnode = graphdb.node(label=data[4], level=***level05***) subbrandlabelindex[data[4] + ***level05***] = tempsubbrandnode temprelationship1 = tempsectornode.OFTYPE(sectornode) temprelationship2 = tempcategorynode.OFTYPE(categorynode) temprelationship3 = tempproductnode.OFTYPE(productnode) temprelationship4 = tempbrandnode.OFTYPE(brandnode) temprelationship5 = tempsubbrandnode.OFTYPE(subbrandnode) temprelationship6 = tempcategorynode.MEMBEROF(tempsectornode) temprelationship7 = tempproductnode.MEMBEROF(tempcategorynode) temprelationship8 = tempbrandnode.MEMBEROF(tempproductnode) temprelationship9 = tempsubbrandnode.MEMBEROF(tempbrandnode) lt_ending_time = datetime.datetime.now() print import Started at : + str(lt_starting_time) + ended at : + str(lt_ending_time) graphdb.shutdown() THX in advance. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] java heep size in neo4j when importing huge data
Hi Francois, When you do things inside of a transaction, it stays in memory until you commit your transaction. If you commit smaller chunks, you won't run into this problem. So instead of doing: li_index = 0 *with graphdb.transaction: for data in data_list:* data[0] = unicode(data[0], 'utf-8') data[1] = unicode(data[1], 'utf-8') data[2] = unicode(data[2], 'utf-8') ... You can break it up in smaller chunks, like this: li_index = 0 *for data in data_list: ** with graphdb.transaction:* data[0] = unicode(data[0], 'utf-8') data[1] = unicode(data[1], 'utf-8') data[2] = unicode(data[2], 'utf-8') ... In the Java API there is a way of doing batch-insertion which is better suited for bulk imports like this, but it is not exposed in neo4j.py. HTH, Andrés On Thu, Sep 9, 2010 at 8:49 AM, Francois Kassis francois_kas...@hotmail.com wrote: hi all, I am trying to import csv data to a new graph database using neo4j.PY after a 65000 row of insertion I am getting a java heep size error knowing that I extended the jvm memory to 640MB. how can I save the data after each 5 row of insertion? how can I use the begintransaction function in python version of neo4j.py? I am using the following script: graphdb = neo4j.GraphDatabase(lparam_database_path, classpath=lparam_kernel_class_path, jvm=lparam_jvm_path, heap_size=ls_jvm_heap_size) with graphdb.transaction: sectornode = Subreference.Node.SECTOR_ROOT(graphdb, label=Sectors, level=1) categorynode = graphdb.node(label=Categories, level=2) productnode = graphdb.node(label=Products, level=3) brandnode = graphdb.node(label=Brands, level=4) subbrandnode = graphdb.node(label=Subbrands, level=5) relationship1 = sectornode.CATEGORY_ROOT(categorynode) relationship2 = categorynode.PRODUCT_ROOT(productnode) relationship3 = productnode.BRAND_ROOT(brandnode) relationship4 = brandnode.SUBBRAND_ROOT(subbrandnode) sectorlabelindex = graphdb.index(sector_label_index, create=True) categorylabelindex = graphdb.index(category_label_index, create=True) productlabelindex = graphdb.index(product_label_index, create=True) brandlabelindex = graphdb.index(brand_label_index, create=True) subbrandlabelindex = graphdb.index(subbrand_label_index, create=True) li_index = 0 with graphdb.transaction: for data in data_list: data[0] = unicode(data[0], 'utf-8') data[1] = unicode(data[1], 'utf-8') data[2] = unicode(data[2], 'utf-8') data[3] = unicode(data[3], 'utf-8') data[4] = unicode(data[4], 'utf-8') li_index = li_index + 1 li_counter = li_counter + 1 ls_message = print importing row + str(li_index) + of + str(li_total_rows) tempsectornode = sectorlabelindex[data[0] + ***level01***] tempcategorynode = categorylabelindex[data[1] + ***level02***] tempproductnode = productlabelindex[data[2] + ***level03***] tempbrandnode = brandlabelindex[data[3] + ***level04***] tempsubbrandnode = subbrandlabelindex[data[4] + ***level05***] try: ls_message = ls_message + tempsectornode[label] + , except: tempsectornode = graphdb.node(label=data[0], level=***level01***) sectorlabelindex[data[0] + ***level01***] = tempsectornode try: ls_message = ls_message + tempcategorynode[label] + , except: tempcategorynode = graphdb.node(label=data[1], level=***level02***) categorylabelindex[data[1] + ***level02***] = tempcategorynode try: ls_message = ls_message + tempproductnode[label] + , except: tempproductnode = graphdb.node(label=data[2], level=***level03***) productlabelindex[data[2] + ***level03***] = tempproductnode try: ls_message = ls_message + tempbrandnode[label] + , except: tempbrandnode = graphdb.node(label=data[3], level=***level04***) brandlabelindex[data[3] + ***level04***] = tempbrandnode try: ls_message = ls_message + tempsubbrandnode[label] + , except: tempsubbrandnode = graphdb.node(label=data[4], level=***level05***) subbrandlabelindex[data[4] + ***level05***] = tempsubbrandnode temprelationship1 = tempsectornode.OFTYPE(sectornode) temprelationship2 = tempcategorynode.OFTYPE(categorynode) temprelationship3 = tempproductnode.OFTYPE(productnode) temprelationship4 = tempbrandnode.OFTYPE(brandnode) temprelationship5 = tempsubbrandnode.OFTYPE(subbrandnode) temprelationship6 = tempcategorynode.MEMBEROF(tempsectornode) temprelationship7 = tempproductnode.MEMBEROF(tempcategorynode) temprelationship8 = tempbrandnode.MEMBEROF(tempproductnode) temprelationship9 = tempsubbrandnode.MEMBEROF(tempbrandnode) lt_ending_time =
Re: [Neo4j] java heep size in neo4j when importing huge data
I tried it but will be extremely slow
Thx anyway.
--
From: user-requ...@lists.neo4j.org
Sent: Thursday, September 09, 2010 1:00 PM
To: user@lists.neo4j.org
Subject: User Digest, Vol 42, Issue 19
Send User mailing list submissions to
user@lists.neo4j.org
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.neo4j.org/mailman/listinfo/user
or, via email, send a message with subject or body 'help' to
user-requ...@lists.neo4j.org
You can reach the person managing the list at
user-ow...@lists.neo4j.org
When replying, please edit your Subject line so it is more specific
than Re: Contents of User digest...
Today's Topics:
1. Re: IndexProvider question (Mattias Persson)
2. Re: erlang-neo4j (Dmitrii Dimandt)
3. Re: erlang-neo4j (Marko Rodriguez)
4. Re: IndexProvider question (Honnur Vorvoi)
5. java heep size in neo4j when importing huge data
(Francois Kassis)
6. Re: java heep size in neo4j when importing huge data
(Andres Taylor)
--
Message: 1
Date: Wed, 8 Sep 2010 16:28:56 +0200
From: Mattias Persson matt...@neotechnology.com
Subject: Re: [Neo4j] IndexProvider question
To: Neo4j user discussions user@lists.neo4j.org
Message-ID:
aanlktin4cjw=smw00=1nlkt8ftmys6xtnvtrve_j9...@mail.gmail.com
Content-Type: text/plain; charset=UTF-8
Hi Honnur!
2010/9/6, Honnur Vorvoi vhon...@yahoo.com:
Hello,
I have the following questions with regard to the IndexProvider(example
below):
1. I already have LuceneFulltextQueryIndexService. Can I use
IndexProvider
with the same graphDb as well? or are they mutually exclusive?
They are separate from one another so both can be used alongside of
each other. Something stored in one of either
LuceneIndexService/LuceneIndexProvider won't affect the other.
2. What doesn the param users in provider.nodeIndex(users) represent?
The LuceneIndexService can only keep values from one key in each
index, but the new LuceneIndexProvider can spawn indexes which can
contain any number of keys and values (making compound queries
possible). Since an index isn't tied to a property key you must name
each index yourself. Each index can also be configured to be either
fulltext or not, to use lower case conversion or not, a.s.o.
3. Do I need to add all the properties in IndexNode(line# 45) in order
to
query? (I have already index the same properties with
LuceneFulltextQueryIndexService)
see my answer for (1), in short: LuceneIndexProvider and the indexes
it spawns has nothing to do with LuceneIndexService (or any derivative
thereof) and hence can't share state.
4. Is it easy to include the query(String) method in
LuceneFulltextQueryIndexService, so I can use just one indexservice
otherwise I would be using LuceneIndexProvider just for query(String)
method.
To add compound querying the storage format (i.e. Lucene usage) needed
to change in incompatible ways, so it isn't an easy fix to add that.
It could however be done by querying multiple indexes in parallell and
merge the results afterwards, but I don't think performance would be
anywhere near using Lucene the right way for compound queries, as
LuceneIndexProvider does.
As alwasy, appreciate your suggestions/recommendations
1 IndexProvider provider = new LuceneIndexProvider( graphDb );
2 IndexNode myIndex = provider.nodeIndex( users );
3
4 myIndex.add( myNode, type, value1 );
5 myIndex.add( myNode, key1, value2 );
6
7 // Ask lucene queries directly here
8 for ( Node searchHit : myIndex.query( type:value1 AND
key1:value2 )
)
9 {
10 System.out.println( Found + searchHit );
11 }
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
--
Mattias Persson, [matt...@neotechnology.com]
Hacker, Neo Technology
www.neotechnology.com
--
Message: 2
Date: Wed, 8 Sep 2010 17:35:44 +0300
From: Dmitrii Dimandt dmit...@dmitriid.com
Subject: Re: [Neo4j] erlang-neo4j
To: gremlin-us...@googlegroups.com
Cc: Neo4j user discussions user@lists.neo4j.org
Message-ID: 083266c3-de36-42a4-ba05-4995a1b64...@gmail.com
Content-Type: text/plain; charset=us-ascii
Yeah, it'll be my next step, I guess :) Blueprints make this stuff
super-easy :) I also see plain simple FS in there as well ;)
BTW, I've also noticed there's mongodb support for storage (?) How would
you use that?
Hi Dimitri,
Great. If you go through Blueprints (as I assume you are), you can also
connect it to OrientDB and RDF SAIL.
Thanks,
Marko.
http://markorodriguez.com
On Sep 8, 2010, at 8:22 AM, Dmitrii Dimandt wrote:
cali has been updated slightly to become database-agnostic.
There are currently connectors to neo4j and TinkerGraph. It's now quite
easy to add your own
Re: [Neo4j] java heep size in neo4j when importing huge data
Hi again Francois,
You could try committing every 100 rows or something like that. The code is
a bit more complex when you can't use the with statement, but it would look
something like this:
try:
tx = graphdb.transaction.begin()
for num, data in enumerate(data_list):
if num % 100:
tx.success()
tx.finish()
tx = graphdb.transaction.begin()
data[0] = unicode(data[0], 'utf-8')
data[1] = unicode(data[1], 'utf-8')
data[2] = unicode(data[2], 'utf-8')
except:
tx.failure()
else:
tx.success()
finally:
tx.finish()
Hth,
Andres
On Thu, Sep 9, 2010 at 12:07 PM, Francois Kassis
francois_kas...@hotmail.com wrote:
I tried it but will be extremely slow
Thx anyway.
--
From: user-requ...@lists.neo4j.org
Sent: Thursday, September 09, 2010 1:00 PM
To: user@lists.neo4j.org
Subject: User Digest, Vol 42, Issue 19
Send User mailing list submissions to
user@lists.neo4j.org
To subscribe or unsubscribe via the World Wide Web, visit
https://lists.neo4j.org/mailman/listinfo/user
or, via email, send a message with subject or body 'help' to
user-requ...@lists.neo4j.org
You can reach the person managing the list at
user-ow...@lists.neo4j.org
When replying, please edit your Subject line so it is more specific
than Re: Contents of User digest...
Today's Topics:
1. Re: IndexProvider question (Mattias Persson)
2. Re: erlang-neo4j (Dmitrii Dimandt)
3. Re: erlang-neo4j (Marko Rodriguez)
4. Re: IndexProvider question (Honnur Vorvoi)
5. java heep size in neo4j when importing huge data
(Francois Kassis)
6. Re: java heep size in neo4j when importing huge data
(Andres Taylor)
--
Message: 1
Date: Wed, 8 Sep 2010 16:28:56 +0200
From: Mattias Persson matt...@neotechnology.com
Subject: Re: [Neo4j] IndexProvider question
To: Neo4j user discussions user@lists.neo4j.org
Message-ID:
aanlktin4cjw=smw00=1nlkt8ftmys6xtnvtrve_j9...@mail.gmail.com
Content-Type: text/plain; charset=UTF-8
Hi Honnur!
2010/9/6, Honnur Vorvoi vhon...@yahoo.com:
Hello,
I have the following questions with regard to the IndexProvider(example
below):
1. I already have LuceneFulltextQueryIndexService. Can I use
IndexProvider
with the same graphDb as well? or are they mutually exclusive?
They are separate from one another so both can be used alongside of
each other. Something stored in one of either
LuceneIndexService/LuceneIndexProvider won't affect the other.
2. What doesn the param users in provider.nodeIndex(users)
represent?
The LuceneIndexService can only keep values from one key in each
index, but the new LuceneIndexProvider can spawn indexes which can
contain any number of keys and values (making compound queries
possible). Since an index isn't tied to a property key you must name
each index yourself. Each index can also be configured to be either
fulltext or not, to use lower case conversion or not, a.s.o.
3. Do I need to add all the properties in IndexNode(line# 45) in
order
to
query? (I have already index the same properties with
LuceneFulltextQueryIndexService)
see my answer for (1), in short: LuceneIndexProvider and the indexes
it spawns has nothing to do with LuceneIndexService (or any derivative
thereof) and hence can't share state.
4. Is it easy to include the query(String) method in
LuceneFulltextQueryIndexService, so I can use just one indexservice
otherwise I would be using LuceneIndexProvider just for query(String)
method.
To add compound querying the storage format (i.e. Lucene usage) needed
to change in incompatible ways, so it isn't an easy fix to add that.
It could however be done by querying multiple indexes in parallell and
merge the results afterwards, but I don't think performance would be
anywhere near using Lucene the right way for compound queries, as
LuceneIndexProvider does.
As alwasy, appreciate your suggestions/recommendations
1 IndexProvider provider = new LuceneIndexProvider( graphDb );
2 IndexNode myIndex = provider.nodeIndex( users );
3
4 myIndex.add( myNode, type, value1 );
5 myIndex.add( myNode, key1, value2 );
6
7 // Ask lucene queries directly here
8 for ( Node searchHit : myIndex.query( type:value1 AND
key1:value2 )
)
9 {
10 System.out.println( Found + searchHit );
11 }
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
--
Mattias Persson, [matt...@neotechnology.com]
Hacker, Neo Technology
www.neotechnology.com
--
Message: 2
Date: Wed, 8 Sep 2010 17:35:44 +0300
From: Dmitrii Dimandt dmit...@dmitriid.com
Subject: Re: [Neo4j] erlang-neo4j
To: gremlin-us...@googlegroups.com
Cc: Neo4j user
Re: [Neo4j] java heep size in neo4j when importing huge data
thx a lot andres it worked fine with ur solution. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] OSGi Dependency Injection
Sounds like a plan ( I intend to touch as little as possible) ! Let me know when you publish something to watch to the public repo or some visible area so we can talk about that. - if you want. Cheers. On Wed, Sep 8, 2010 at 1:36 PM, Andreas Kollegger andreas.kolleg...@neotechnology.com wrote: Hey Toni! Thanks, I'm excited to be working with the team here. They're a great bunch of guys, and of course I love the graph stuff. Though we haven't met in person, your name is very familiar to me wherever OSGi conversations come up. It's great to know you follow the neo4j threads. While I'm just lifting the hood to have a look around the engine, your assumptions seem correct. The neo4j kernel has a notion of extensions which it currently attempts to load through java ServiceLoaders, or a custom loader implementation. So it has a dual nature of being the persistence service and the runtime platform upon which everything is built. It's a common situation where, as you suggest, OSGi is a perfect fit for the general solution. For now, though, I intend to touch as little as possible. I'll have to make some subtle changes to the kernel startup to enable it to be OSGi aware, but the changes should be invisible in non-OSGi environments. Cheers, Andreas On Wed, Sep 8, 2010 at 12:42 PM, Toni Menzel t...@okidokiteam.com wrote: Hi Andreas, glad to see you in Neo Technology and directly grabbing the OSGi topic. Even if you don't know me, but have heard from Peter about you working on that part soon'ish. I am also pretty much into OSGi and try to spread the love wherever i can. So i try to rephrase the assumptions that arise from your writup because i don't know all parts of neo4j yet myself. I know that there are things like the different indexers (lucene), RDF representations, some graph algo stuff and so on. Is this what you mean by Extensions in the Neo context ? OSGi-friendly kernel extension loader means having whiteboard pattern api that does not depend on OSGi right? So, in an osgi runtime you would register all the extensions (either they do themselves with activator code, or you get rid of this too cause they are boilerplate). Then there is a kernel OSGi service which is basically the only OSGi aware Neo4J component. I stacks the neo runtime together and takes care of the dynamic nature. Outside of OSGi you have a much simpler, static runtime (maybe the one that is already in place). So, i don't know how it really looks like now, even don't know what roles extensions can play for the neo4j runtime. Another thing arise, may a future thought, once the dynamic runtime gets stable, to make the default. See, you probably know, OSGi is not the fat container that you need to mock out if you want it small. Just boot an embedded felix or use concierge which is really small. It in the end really gets about what role the extensions will/can play. If they are really a vital part, if neo is a composition of services, then going full osgi can be a natural destination. Anyway, i put in some thoughts and assumptions, hope its not totally off the point. cheers, Toni -- *Toni Menzel || **http://okidokiteam.com* On Wed, Sep 8, 2010 at 11:36 AM, Andreas Kollegger andreas.kolleg...@neotechnology.com wrote: Hi all, I have a proposal to make regarding the use of Neo4j under OSGi. All the components currently build as proper, deployable OSGi bundles, which is great. I'd like to introduce an OSGi-friendly kernel extension loader, establishing a general patten for including extensions to any component that is not very intrusive. There should never be any dependencies from a component outwards to OSGi, the dependencies should point from OSGi inwards. So, I'd like to add OSGi bundle activators which would only get referenced when starting the bundle in an OSGi environment. For the kernel, the activator would inject a kernel extension loader into the kernel. The extension loader would listen for advertised kernel extension services, which would consequently get loaded when a new graph db is created. Generally, something like the attached UML. While I think this is a reasonable first step, the dynamic nature of the OSGi runtime means that there could be problems within the kernel if it uses stale references to extensions. Those will have to be addressed as they come up. Thoughts? -Andreas ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Using the REST neo4j
On Tuesday, September 7, 2010, Jacob Hansson ja...@voltvoodoo.com wrote: 2010/9/6 Alexandru Popescu ☀ the.mindstorm.mailingl...@gmail.com On Monday, September 6, 2010, Jim Webber j...@webber.name wrote: Hi Alex, While I still can achieve all these with the current packaging, it feels more hacky: I need to create a new Jetty6BasedWebServer or modify the existing one to enhance it with my own stuff. Each change would require compiling and repackaging the whole neo4j-rest. Definitely not as easy as dropping in my own jar and a new web.xml. That's an interesting point. In a sense, the neo-rest package is Neo's REST package. Interesting... My main question is: what exactly is this package offering to the end user in the current form? IMO it cannot be an off-the-shelf product as there is no security. It is not a library either, as extending it is not so easy. Basically, and without any intention to harm any feelings, it looks like one of those dummy web UI interface to X. And I'd say it has much more potential than that! Jacob, I must confess I'm totally confused by your comments below. I've always seen it as the beginnings of a proper stand-alone neo4j server. If it is the beginning, then what comes next? And more importantly from whom? Basically my proposal was meant to make things easier for people to built on top of it, so I'm not really sure how you see the continuation of it. A REST/JSON API to Neo4j opens up for remote clients in any language, and would be an important part in matching offerings from other database vendors. While extendability is a great thing, building it as a library and/or packaging it as a WAR makes it very java-centric. Currently the neo4j-rest is distributed as a java application. So it is java-centric. What makes it attractive is that it allows using the HTTP protocol. Providing neo4j-rest as both a self contained app and as a web app will give you exactly the same benefits, with additional freedom on choosing how to use it, what servers to deploy it too, etc. Like you say, there is no security, and I agree it is currently the main culprit stopping neo4j REST from production use. This can of course be offset with firewalling etc, but I couldn't agree more of the importance of a proper security layer. Security was used as a basic example of things that could be much easier to be added on top of the neo4j-rest if provided in a simpler format. As you probably know already firewalls will give you at most a very basic sort of authentication, but nothing else. As far as UI interface to X goes, the area to focus on I think is the JSON part of the API. With that, a UI can be built in any language. Take a look at http://github.com/neo4j/webadmin for a more powerful browsing UI for neo4j REST. I think you mis-read my post. I'm not looking for a nice UI, but rather for a basis to further build REST services on top of a neo4j db. As Jim mentioned in his posts, currently neo4j-rest is just exposing the basics of a neo4j db. :- alex However the notion of just letting end users write their own code hadn't occurred at least to me. I guess I always assumed that if users really wanted a domain specific API then they'd write their own. But the notion of user-registered filters (at least) is pretty sensible. I think your initial assumption makes a lot of sense. But why would one have to duplicate all the work when this could provide him not only with a good example, but a common basis for a complete solution. I'm looking at it from the perspective of a DB vizualization tool: what's in there offers you the default view. Next you could build your own views, etc. You could even build your complete application using it. Best thing is that I don't even think it is difficult to get it being more a matter of packaging than anything else. All would be needed: - a web.xml file with some configuration options in it (db location) - providing better access to the GraphDatabaseService (see my previous suggestion) and other common shared resources (IndexService, etc) - a different final package in form of a war - done Does it make sense to you? :- alex Jim ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Jacob Hansson Phone: +46 (0) 763503395 Twitter: @jakewins ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] IndexProvider question
Thanks Mattias.
Since IndexProvider does all LuceneFulltextQueryIndexService can do and much
more, I am going to use just IndexProvider.
Date: Wed, 8 Sep 2010 16:28:56 +0200
From: Mattias Persson matt...@neotechnology.com
Subject: Re: [Neo4j] IndexProvider question
To: Neo4j user discussions user@lists.neo4j.org
Message-ID:
aanlktin4cjw=smw00=1nlkt8ftmys6xtnvtrve_j9...@mail.gmail.com
Content-Type: text/plain; charset=UTF-8
Hi Honnur!
2010/9/6, Honnur Vorvoi vhon...@yahoo.com:
Hello,
I have the following questions with regard to the IndexProvider(example
below):
1. I already have LuceneFulltextQueryIndexService. Can I use IndexProvider
with the same graphDb as well? or are they mutually exclusive?
They are separate from one another so both can be used alongside of
each other. Something stored in one of either
LuceneIndexService/LuceneIndexProvider won't affect the other.
2. What doesn the param users in provider.nodeIndex(users) represent?
The LuceneIndexService can only keep values from one key in each
index, but the new LuceneIndexProvider can spawn indexes which can
contain any number of keys and values (making compound queries
possible). Since an index isn't tied to a property key you must name
each index yourself. Each index can also be configured to be either
fulltext or not, to use lower case conversion or not, a.s.o.
3. Do I need to add all the properties in IndexNode(line# 45) in order to
query? (I have already index the same properties with
LuceneFulltextQueryIndexService)
see my answer for (1), in short: LuceneIndexProvider and the indexes
it spawns has nothing to do with LuceneIndexService (or any derivative
thereof) and hence can't share state.
4. Is it easy to include the query(String) method in
LuceneFulltextQueryIndexService, so I can use just one indexservice
otherwise I would be using LuceneIndexProvider just for query(String)
method.
To add compound querying the storage format (i.e. Lucene usage) needed
to change in incompatible ways, so it isn't an easy fix to add that.
It could however be done by querying multiple indexes in parallell and
merge the results afterwards, but I don't think performance would be
anywhere near using Lucene the right way for compound queries, as
LuceneIndexProvider does.
As alwasy, appreciate your suggestions/recommendations
1 IndexProvider provider = new LuceneIndexProvider( graphDb );
2 IndexNode myIndex = provider.nodeIndex( users );
3
4 myIndex.add( myNode, type, value1 );
5 myIndex.add( myNode, key1, value2 );
6
7 // Ask lucene queries directly here
8 for ( Node searchHit : myIndex.query( type:value1 AND key1:value2 )
)
9 {
10 System.out.println( Found + searchHit );
11 }
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
--
Mattias Persson, [matt...@neotechnology.com]
Hacker, Neo Technology
www.neotechnology.com
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
