That easy, huh? Heck, this gets better and better. BTW, how about escaping?
Dennis Gearon Signature Warning ---------------- It is always a good idea to learn from your own mistakes. It is usually a better idea to learn from others’ mistakes, so you do not have to make them yourself. from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036' EARTH has a Right To Life, otherwise we all die. ----- Original Message ---- From: Adam Estrada <estrada.adam.gro...@gmail.com> To: Dennis Gearon <gear...@sbcglobal.net>; solr-user@lucene.apache.org Sent: Thu, December 16, 2010 10:58:47 AM Subject: Re: bulk commits This is how I import a lot of data from a cvs file. There are close to 100k records in there. Note that you can either pre-define the column names using the fieldnames param like I did here *or* include header=true which will automatically pick up the column header if your file has it. curl " http://localhost:8983/solr/update/csv?commit=true&separator=%2C&fieldnames=id,name,asciiname,lat,lng,countrycode,population,elevation,gtopo30,timezone,modificationdate,cat&stream.file=C :\tmp\cities1000.csv&overwrite=true&stream.contentType=text/plain;charset=utf-8" This seems to load everything in to some kind of temporary location before it's actually committed. If something goes wrong there is a rollback feature that will undo anything that happened before the commit. As far as batching a bunch of files, I copied and pasted the following in to Cygwin and it worked just fine. curl " http://localhost:8983/solr/update/csv?commit=true&separator=%2C&fieldnames=id,name,asciiname,lat,lng,countrycode,population,elevation,gtopo30,timezone,modificationdate,cat&stream.file=C :\tmp\cities1000.csv&overwrite=true&stream.contentType=text/plain;charset=utf-8" curl " http://localhost:8983/solr/update/csv?commit=true&separator=%2C&fieldnames=id,name,asciiname,latitude,longitude,featureclass,featurecode,countrycode,admin1code,admin2code,admin3code,admin4code,population,elevation,gtopo30,timezone,modificationdate&stream.file=C :\tmp\xab.csv&overwrite=true&stream.contentType=text/plain;charset=utf-8" curl " http://localhost:8983/solr/update/csv?commit=true&separator=%2C&fieldnames=id,name,asciiname,latitude,longitude,featureclass,featurecode,countrycode,admin1code,admin2code,admin3code,admin4code,population,elevation,gtopo30,timezone,modificationdate&stream.file=C :\tmp\xac.csv&overwrite=true&stream.contentType=text/plain;charset=utf-8" curl " http://localhost:8983/solr/update/csv?commit=true&separator=%2C&fieldnames=id,name,asciiname,latitude,longitude,featureclass,featurecode,countrycode,admin1code,admin2code,admin3code,admin4code,population,elevation,gtopo30,timezone,modificationdate&stream.file=C :\tmp\xad.csv&overwrite=true&stream.contentType=text/plain;charset=utf-8" curl " http://localhost:8983/solr/update/csv?commit=true&separator=%2C&fieldnames=id,name,asciiname,latitude,longitude,featureclass,featurecode,countrycode,admin1code,admin2code,admin3code,admin4code,population,elevation,gtopo30,timezone,modificationdate&stream.file=C :\tmp\xae.csv&overwrite=true&stream.contentType=text/plain;charset=utf-8" curl " http://localhost:8983/solr/update/csv?commit=true&separator=%2C&fieldnames=id,name,asciiname,latitude,longitude,featureclass,featurecode,countrycode,admin1code,admin2code,admin3code,admin4code,population,elevation,gtopo30,timezone,modificationdate&stream.file=C :\tmp\xaf.csv&overwrite=true&stream.contentType=text/plain;charset=utf-8" curl " http://localhost:8983/solr/update/csv?commit=true&separator=%2C&fieldnames=id,name,asciiname,latitude,longitude,featureclass,featurecode,countrycode,admin1code,admin2code,admin3code,admin4code,population,elevation,gtopo30,timezone,modificationdate&stream.file=C :\tmp\xag.csv&overwrite=true&stream.contentType=text/plain;charset=utf-8" curl " http://localhost:8983/solr/update/csv?commit=true&separator=%2C&fieldnames=id,name,asciiname,latitude,longitude,featureclass,featurecode,countrycode,admin1code,admin2code,admin3code,admin4code,population,elevation,gtopo30,timezone,modificationdate&stream.file=C :\tmp\xah.csv&overwrite=true&stream.contentType=text/plain;charset=utf-8" curl " http://localhost:8983/solr/update/csv?commit=true&separator=%2C&fieldnames=id,name,asciiname,latitude,longitude,featureclass,featurecode,countrycode,admin1code,admin2code,admin3code,admin4code,population,elevation,gtopo30,timezone,modificationdate&stream.file=C :\tmp\xai.csv&overwrite=true&stream.contentType=text/plain;charset=utf-8" curl " http://localhost:8983/solr/update/csv?commit=true&separator=%2C&fieldnames=id,name,asciiname,latitude,longitude,featureclass,featurecode,countrycode,admin1code,admin2code,admin3code,admin4code,population,elevation,gtopo30,timezone,modificationdate&stream.file=C :\tmp\xaj.csv&overwrite=true&stream.contentType=text/plain;charset=utf-8" curl " http://localhost:8983/solr/update/csv?commit=true&separator=%2C&fieldnames=id,name,asciiname,latitude,longitude,featureclass,featurecode,countrycode,admin1code,admin2code,admin3code,admin4code,population,elevation,gtopo30,timezone,modificationdate&stream.file=C :\tmp\xak.csv&overwrite=true&stream.contentType=text/plain;charset=utf-8" curl " http://localhost:8983/solr/update/csv?commit=true&separator=%2C&fieldnames=id,name,asciiname,latitude,longitude,featureclass,featurecode,countrycode,admin1code,admin2code,admin3code,admin4code,population,elevation,gtopo30,timezone,modificationdate&stream.file=C :\tmp\xal.csv&overwrite=true&stream.contentType=text/plain;charset=utf-8" curl " http://localhost:8983/solr/update/csv?commit=true&separator=%2C&fieldnames=id,name,asciiname,latitude,longitude,featureclass,featurecode,countrycode,admin1code,admin2code,admin3code,admin4code,population,elevation,gtopo30,timezone,modificationdate&stream.file=C :\tmp\xam.csv&overwrite=true&stream.contentType=text/plain;charset=utf-8" curl " http://localhost:8983/solr/update/csv?commit=true&separator=%2C&fieldnames=id,name,asciiname,latitude,longitude,featureclass,featurecode,countrycode,admin1code,admin2code,admin3code,admin4code,population,elevation,gtopo30,timezone,modificationdate&stream.file=C :\tmp\xan.csv&overwrite=true&stream.contentType=text/plain;charset=utf-8" curl " http://localhost:8983/solr/update/csv?commit=true&separator=%2C&fieldnames=id,name,asciiname,latitude,longitude,featureclass,featurecode,countrycode,admin1code,admin2code,admin3code,admin4code,population,elevation,gtopo30,timezone,modificationdate&stream.file=C :\tmp\xao.csv&overwrite=true&stream.contentType=text/plain;charset=utf-8" curl " http://localhost:8983/solr/update/csv?commit=true&separator=%2C&fieldnames=id,name,asciiname,latitude,longitude,featureclass,featurecode,countrycode,admin1code,admin2code,admin3code,admin4code,population,elevation,gtopo30,timezone,modificationdate&stream.file=C :\tmp\xap.csv&overwrite=true&stream.contentType=text/plain;charset=utf-8" curl http://localhost:8983/solr/update -H "Content-Type: text/xml" --data-binary '<optimize/>' Adam On Thu, Dec 16, 2010 at 1:44 PM, Dennis Gearon <gear...@sbcglobal.net>wrote: > Might be Csv or tab delimited text. > > Sent from Yahoo! Mail on Android > > ------------------------------ > * From: * Adam Estrada <estrada.adam.gro...@gmail.com>; > * To: * <solr-user@lucene.apache.org>; > * Subject: * Re: bulk commits > * Sent: * Thu, Dec 16, 2010 6:35:17 PM > > what is it that you are trying to commit? > > a > > On Thu, Dec 16, 2010 at 1:03 PM, Dennis Gearon <gear...@sbcglobal.net > >wrote: > > > What have people found as the best way to do bulk commits either from the > > web or > > from a file on the system? > > > > Dennis Gearon > > > > > > Signature Warning > > ---------------- > > It is always a good idea to learn from your own mistakes. It is usually a > > better > > idea to learn from others’ mistakes, so you do not have to make them > > yourself. > > from 'http://blogs.techrepublic.com.com/security/?p=4501&tag=nl.e036' > > > > > > EARTH has a Right To Life, > > otherwise we all die. > > > > >
