Re: [Dspace-tech] Log into DSpace with curl
Hi Simon, this is new to me as well, I got it to work using: curl --data login_email=myname%40gmail.comlogin_password=passwordsubmit=Sign+in http://www.mydspace.com/password-login I had some issues first using the --data-urlencode parameter instead of --data. This also encoded the ampersands which shouldn't have happened to make it work. If you debug your query, following bit in the return html is interesting: input id=aspect_eperson_PasswordLogin_field_login_email class=ds-text-field error name=login_email type=text value=*contents of your query* title=The email address and/or password supplied were not valid. / span class=error* The email address and/or password supplied were not valid./span because I saw that all my parameters ended up in that field, I realized there was something wrong with the ampersand encoding. On a sidenote, if you play around with forms, encoding and parameters, Tamper Data is a nice little firefox add-on that allows you to see and tamper with the requests you make responses you get. https://addons.mozilla.org/en-us/firefox/addon/tamper-data/ cheers, Bram -- [image: logo] *Bram Luyten* *@mire* *2888 Loker Avenue East, Suite 305, Carlsbad, CA. 92010* *Esperantolaan 4, Heverlee 3001, Belgium* http://www.atmire.com/www.atmire.comhttp://atmire.com/website/?q=servicesutm_source=emailfooterutm_medium=emailutm_campaign=braml On Wed, Aug 1, 2012 at 6:04 PM, Brown, Simon Contractor, Digital Consulting Services scbr...@nps.edu wrote: We are using DSpace 1.8.2 with the JSPUI admin interface. I would like to log into DSpace using curl (or more precisely RCurl) because I want to be able to scrape item IDs (not the handles, but the actual IDs). My code looks something like this: ** ** postForm(“dspace/…./password-login”, login_email=”user”, login_password=”password”) ** ** It basically always returns “The e-mail address and password you supplied were not valid” ** ** Is it not possible to log in using curl? If it is possible I’ll figure it out eventually, but if it’s not I would rather not spend the time on it.** ** ** ** Simon Brown Digital Consulting Services Supporting the Dudley Knox Library, Naval postgraduate School scbr...@nps.edu 831.656.3574 Phone 831.656.3238 Fax ** ** -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Log into DSpace with curl
Hi Simon, as an alternative to screen scraping, you may want to take a look at the REST API [1] which is made exactly for tasks like this one. Another alternative is a simple SQL query [2]. Just ask if you need more details. [1] https://wiki.duraspace.org/display/DSPACE/REST+API [2] https://wiki.duraspace.org/display/DSPACE/DSpace+2.0+Requirements+and+Issues#DSpace2.0RequirementsandIssues-DatabaseOverviewGraphic Regards, ~~helix84 -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Log into DSpace with curl
Yet another alternative could be using Solr directly. Info on how to access Solr is here: https://wiki.duraspace.org/display/DSPACE/Solr The key you'd use is search.resourceid. Regards, ~~helix84 -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Log into DSpace with curl
Hi, Simon, I know that I would be interested in hearing more about what it is you're up to. :-) What's your use case for the item ID once you scrape it? Is this something that the REST-API module might be able to do for you? -- HARDY POTTINGER pottinge...@umsystem.edu University of Missouri Library Systems http://lso.umsystem.edu/~pottingerhj/ https://MOspace.umsystem.edu/ And remember, also added the Princesss of Sweet Rhyme, that many places you would like to see are just off the Map and many things you want to know are just out of sight or a little beyond your reach. But someday you'll reach them after all, for what you learn today, for no reason at all, will help you discover all the wonderful secrets of tomorrow. --Norton Juster, The Phantom Tollbooth On 8/2/12 3:37 AM, Bram Luyten b...@mire.be wrote: Hi Simon, this is new to me as well, I got it to work using: curl --data login_email=myname%40gmail.com http://40gmail.comlogin_password=passwordsubmit=Sign+in http://www.mydspace.com/password-login I had some issues first using the --data-urlencode parameter instead of --data. This also encoded the ampersands which shouldn't have happened to make it work. If you debug your query, following bit in the return html is interesting: input id=aspect_eperson_PasswordLogin_field_login_email class=ds-text-field error name=login_email type=text value=contents of your query title=The email address and/or password supplied were not valid. / span class=error* The email address and/or password supplied were not valid./span because I saw that all my parameters ended up in that field, I realized there was something wrong with the ampersand encoding. On a sidenote, if you play around with forms, encoding and parameters, Tamper Data is a nice little firefox add-on that allows you to see and tamper with the requests you make responses you get. https://addons.mozilla.org/en-us/firefox/addon/tamper-data/ cheers, Bram -- Bram Luyten@mire 2888 Loker Avenue East, Suite 305, Carlsbad, CA. 92010 Esperantolaan 4, Heverlee 3001, Belgium http://www.atmire.com/www.atmire.com http://atmire.com/website/?q=servicesutm_source=emailfooterutm_medium=e mailutm_campaign=braml On Wed, Aug 1, 2012 at 6:04 PM, Brown, Simon Contractor, Digital Consulting Services scbr...@nps.edu wrote: We are using DSpace 1.8.2 with the JSPUI admin interface. I would like to log into DSpace using curl (or more precisely RCurl) because I want to be able to scrape item IDs (not the handles, but the actual IDs). My code looks something like this: postForm(³dspace/Š./password-login², login_email=²user², login_password=²password²) It basically always returns ³The e-mail address and password you supplied were not valid² Is it not possible to log in using curl? If it is possible I¹ll figure it out eventually, but if it¹s not I would rather not spend the time on it. Simon Brown Digital Consulting Services Supporting the Dudley Knox Library, Naval postgraduate School scbr...@nps.edu 831.656.3574 Phone 831.656.3238 Fax -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Log into DSpace with curl
Have you successfully used the 'content searching' part of the API? Didn’t work for me -- very basic queries would fail to return any results. It also lacks any specificity -- how do you do an advanced query with the API? Can you search by author and then limit to a certain publication year??? Probably not, but I can do that with screen scraping. I would love it if the API could do a /search?author=Simon+Brownpubyear=2012 but I don't think it does that? -Original Message- From: ivan.ma...@gmail.com [mailto:ivan.ma...@gmail.com] On Behalf Of helix84 Sent: Thursday, August 02, 2012 2:41 AM To: Brown, Simon Contractor, Digital Consulting Services Cc: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Log into DSpace with curl Hi Simon, as an alternative to screen scraping, you may want to take a look at the REST API [1] which is made exactly for tasks like this one. Another alternative is a simple SQL query [2]. Just ask if you need more details. [1] https://wiki.duraspace.org/display/DSPACE/REST+API [2] https://wiki.duraspace.org/display/DSPACE/DSpace+2.0+Requirements+and+Issues#DSpace2.0RequirementsandIssues-DatabaseOverviewGraphic Regards, ~~helix84 -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
Re: [Dspace-tech] Log into DSpace with curl
My use for this is making a tool for generating/extracting metadata reports (e.g. a bibliography based on a advanced search query) into a useable data format (.csv) for the staff at my library. DSpace lacks functions that allow saving lists and exporting citations so this is kind of a work around. Also, you can extract metadata by collection for batch editing and mapping, but not by query. What if you want to map an author's publications to a new collection for that author but the author's publications are spread across many different collections? I should be able to generate the metadata spreadsheet for batch editing not just based on collection, but based on any query that I can think of doing based on the metadata and full text. I already have a good start to doing this with screen scraping, but I'm just lacking the item IDs -- hence the login with RCurl. This tool is for library staff (and anyone else who is willing), not for the administrator of the website who can do SQL queries if they wanted. People who need to access the data, not just the documents. This is what I have so far (also contains some scripts for making SIPs from a spreadsheet): https://wiki.nps.edu/display/CIR/CalhounTools+--+An+R+Package+for+Common+Repository+Tasks -Original Message- From: Pottinger, Hardy J. [mailto:pottinge...@umsystem.edu] Sent: Thursday, August 02, 2012 7:27 AM To: Bram Luyten; Brown, Simon Contractor, Digital Consulting Services Cc: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Log into DSpace with curl Hi, Simon, I know that I would be interested in hearing more about what it is you're up to. :-) What's your use case for the item ID once you scrape it? Is this something that the REST-API module might be able to do for you? -- HARDY POTTINGER pottinge...@umsystem.edu University of Missouri Library Systems http://lso.umsystem.edu/~pottingerhj/ https://MOspace.umsystem.edu/ And remember, also added the Princesss of Sweet Rhyme, that many places you would like to see are just off the Map and many things you want to know are just out of sight or a little beyond your reach. But someday you'll reach them after all, for what you learn today, for no reason at all, will help you discover all the wonderful secrets of tomorrow. --Norton Juster, The Phantom Tollbooth On 8/2/12 3:37 AM, Bram Luyten b...@mire.be wrote: Hi Simon, this is new to me as well, I got it to work using: curl --data login_email=myname%40gmail.com http://40gmail.comlogin_password=passwordsubmit=Sign+in http://www.mydspace.com/password-login I had some issues first using the --data-urlencode parameter instead of --data. This also encoded the ampersands which shouldn't have happened to make it work. If you debug your query, following bit in the return html is interesting: input id=aspect_eperson_PasswordLogin_field_login_email class=ds-text-field error name=login_email type=text value=contents of your query title=The email address and/or password supplied were not valid. / span class=error* The email address and/or password supplied were not valid./span because I saw that all my parameters ended up in that field, I realized there was something wrong with the ampersand encoding. On a sidenote, if you play around with forms, encoding and parameters, Tamper Data is a nice little firefox add-on that allows you to see and tamper with the requests you make responses you get. https://addons.mozilla.org/en-us/firefox/addon/tamper-data/ cheers, Bram -- Bram Luyten@mire 2888 Loker Avenue East, Suite 305, Carlsbad, CA. 92010 Esperantolaan 4, Heverlee 3001, Belgium http://www.atmire.com/www.atmire.com http://atmire.com/website/?q=servicesutm_source=emailfooterutm_mediu m=e mailutm_campaign=braml On Wed, Aug 1, 2012 at 6:04 PM, Brown, Simon Contractor, Digital Consulting Services scbr...@nps.edu wrote: We are using DSpace 1.8.2 with the JSPUI admin interface. I would like to log into DSpace using curl (or more precisely RCurl) because I want to be able to scrape item IDs (not the handles, but the actual IDs). My code looks something like this: postForm(³dspace/Š./password-login², login_email=²user², login_password=²password²) It basically always returns ³The e-mail address and password you supplied were not valid² Is it not possible to log in using curl? If it is possible I¹ll figure it out eventually, but if it¹s not I would rather not spend the time on it. Simon Brown Digital Consulting Services Supporting the Dudley Knox Library, Naval postgraduate School scbr...@nps.edu 831.656.3574 Phone 831.656.3238 Fax --- --- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats
Re: [Dspace-tech] Log into DSpace with curl
Aha, you may be interested in the SkylighUI project: https://github.com/skylightui/skylight It's a CodeIgnitor (a simple PHP framework) interface which relies upon the DSpace Discovery Solr index. You could just query Solr directly, but I think you might like to borrow some of the heavy lifting in SkylightUI. Kim Shepherd would be able to provide you with more info on the project. -- HARDY POTTINGER pottinge...@umsystem.edu University of Missouri Library Systems http://lso.umsystem.edu/~pottingerhj/ https://MOspace.umsystem.edu/ I am always doing that which I cannot do, in order that I may learn how to do it. --Pablo Picasso On 8/2/12 10:25 AM, Brown, Simon Contractor, Digital Consulting Services scbr...@nps.edu wrote: My use for this is making a tool for generating/extracting metadata reports (e.g. a bibliography based on a advanced search query) into a useable data format (.csv) for the staff at my library. DSpace lacks functions that allow saving lists and exporting citations so this is kind of a work around. Also, you can extract metadata by collection for batch editing and mapping, but not by query. What if you want to map an author's publications to a new collection for that author but the author's publications are spread across many different collections? I should be able to generate the metadata spreadsheet for batch editing not just based on collection, but based on any query that I can think of doing based on the metadata and full text. I already have a good start to doing this with screen scraping, but I'm just lacking the item IDs -- hence the login with RCurl. This tool is for library staff (and anyone else who is willing), not for the administrator of the website who can do SQL queries if they wanted. People who need to access the data, not just the documents. This is what I have so far (also contains some scripts for making SIPs from a spreadsheet): https://wiki.nps.edu/display/CIR/CalhounTools+--+An+R+Package+for+Common+R epository+Tasks -Original Message- From: Pottinger, Hardy J. [mailto:pottinge...@umsystem.edu] Sent: Thursday, August 02, 2012 7:27 AM To: Bram Luyten; Brown, Simon Contractor, Digital Consulting Services Cc: dspace-tech@lists.sourceforge.net Subject: Re: [Dspace-tech] Log into DSpace with curl Hi, Simon, I know that I would be interested in hearing more about what it is you're up to. :-) What's your use case for the item ID once you scrape it? Is this something that the REST-API module might be able to do for you? -- HARDY POTTINGER pottinge...@umsystem.edu University of Missouri Library Systems http://lso.umsystem.edu/~pottingerhj/ https://MOspace.umsystem.edu/ And remember, also added the Princesss of Sweet Rhyme, that many places you would like to see are just off the Map and many things you want to know are just out of sight or a little beyond your reach. But someday you'll reach them after all, for what you learn today, for no reason at all, will help you discover all the wonderful secrets of tomorrow. --Norton Juster, The Phantom Tollbooth On 8/2/12 3:37 AM, Bram Luyten b...@mire.be wrote: Hi Simon, this is new to me as well, I got it to work using: curl --data login_email=myname%40gmail.com http://40gmail.comlogin_password=passwordsubmit=Sign+in http://www.mydspace.com/password-login I had some issues first using the --data-urlencode parameter instead of --data. This also encoded the ampersands which shouldn't have happened to make it work. If you debug your query, following bit in the return html is interesting: input id=aspect_eperson_PasswordLogin_field_login_email class=ds-text-field error name=login_email type=text value=contents of your query title=The email address and/or password supplied were not valid. / span class=error* The email address and/or password supplied were not valid./span because I saw that all my parameters ended up in that field, I realized there was something wrong with the ampersand encoding. On a sidenote, if you play around with forms, encoding and parameters, Tamper Data is a nice little firefox add-on that allows you to see and tamper with the requests you make responses you get. https://addons.mozilla.org/en-us/firefox/addon/tamper-data/ cheers, Bram -- Bram Luyten@mire 2888 Loker Avenue East, Suite 305, Carlsbad, CA. 92010 Esperantolaan 4, Heverlee 3001, Belgium http://www.atmire.com/www.atmire.com http://atmire.com/website/?q=servicesutm_source=emailfooterutm_mediu m=e mailutm_campaign=braml On Wed, Aug 1, 2012 at 6:04 PM, Brown, Simon Contractor, Digital Consulting Services scbr...@nps.edu wrote: We are using DSpace 1.8.2 with the JSPUI admin interface. I would like to log into DSpace using curl (or more precisely RCurl) because I want to be able to scrape item IDs (not the handles, but the actual IDs). My code looks something like this: postForm(³dspace/Š./password-login², login_email=²user², login_password=²password²) It basically always returns ³The e-mail
Re: [Dspace-tech] Log into DSpace with curl
On Thu, Aug 2, 2012 at 5:12 PM, Brown, Simon Contractor, Digital Consulting Services scbr...@nps.edu wrote: Have you successfully used the 'content searching' part of the API? Didn’t work for me -- very basic queries would fail to return any results. It also lacks any specificity -- how do you do an advanced query with the API? Can you search by author and then limit to a certain publication year??? Probably not, but I can do that with screen scraping. I would love it if the API could do a /search?author=Simon+Brownpubyear=2012 but I don't think it does that? I haven't tested this aspect of the REST API, so I really don't know. The documentation is also very silent on this point [1] - doesn't say anything on what query parameter should look like. However, Solr should be perfect for your purpose. Any query you can express in Discovery you can ask Solr directly. You may be also interested in these developments: [2] [3] [2] Doesn't have the code published yet, but it's coming. It would be very helpful if you could review and comment on [3]. [1] https://wiki.duraspace.org/display/DSPACE/REST+API#RESTAPI-Contentsearching [2] https://jira.duraspace.org/browse/DS-1217 [3] https://jira.duraspace.org/browse/DS-1224 Regards, ~~helix84 -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/ ___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
[Dspace-tech] Log into DSpace with curl
We are using DSpace 1.8.2 with the JSPUI admin interface. I would like to log into DSpace using curl (or more precisely RCurl) because I want to be able to scrape item IDs (not the handles, but the actual IDs). My code looks something like this: postForm(dspace//password-login, login_email=user, login_password=password) It basically always returns The e-mail address and password you supplied were not valid Is it not possible to log in using curl? If it is possible I'll figure it out eventually, but if it's not I would rather not spend the time on it. Simon Brown Digital Consulting Services Supporting the Dudley Knox Library, Naval postgraduate School scbr...@nps.edu 831.656.3574 Phone 831.656.3238 Fax -- Live Security Virtual Conference Exclusive live event will cover all the ways today's security and threat landscape has changed and how IT managers can respond. Discussions will include endpoint security, mobile security and the latest in malware threats. http://www.accelacomm.com/jaw/sfrnl04242012/114/50122263/___ DSpace-tech mailing list DSpace-tech@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/dspace-tech
