Re: TypeError when calling oaiharvest from CLI

2015-08-12 Thread Werner Greßhoff 

Hi,

I tried this now and it started well, meaning the harvesting process
started. After some time the oaiharvest process changed status from
"RUNNING" to "ERRORS REPO" after approx. 90-100 seconds but after
a look in the process list it seems to be still running (even after
deleting it from bibsched).

Best regards,
Werner


Am 07.08.2015 um 15:53 schrieb Jan Aage Lavik:

Hi,

I see the problem now. Unfortunately, with the changes in 2.0 not all
parts of the legacy admin UI was fixed. The in browser display of
harvests is one of those. Instead, you can use the `oaiharvest`
command-line tool to schedule a harvest in `bibsched`, our legacy task
scheduler.

E.g.

$ oaiharvest -r desy —dates=-mm-dd:-mm-dd# to avoid
harvesting the entire repo

Then start bibched:

$ bibsched start

You can monitor the process by accessing bibsched:

$ bibsched

Press ‘h’ for a list of help commands.

To be able to fully run the harvest you also need to make sure a Celery
worker instance is running as described here:
http://invenio.readthedocs.org/en/latest/getting-started/first-steps.html?highlight=celery#development

This will execute a process which will process the harvested data and
then finally create bibupload jobs that you can run in bibsched (if you
tick off upload step in post-process).

Word of warning, though, this version of Invenio is a developer preview
release and not fully stable. We are working hard towards the next
release to address these and other issues.

Hope this helps.

Cheers,
Jan


On 07 Aug 2015, at 15:25, Werner Greßhoff
mailto:werner.gressh...@uni-muenster.de>> wrote:

Hello!

First things first: changing the workflow to
'oaiharvest_harvest_repositories' doesn't change the result!
So here is now the log:

2015-08-07 15:18:05,935 ERROR: /js/jquery.min.js [in
/home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py:142]
Traceback (most recent call last):
 File
"/home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py",
line 137, in page_not_found
   response = legacy_application(request.environ, g.start_response)
 File
"/home/system/.virtualenvs/invenio/src/invenio/invenio/legacy/wsgi/__init__.py",
line 485, in application
   raise NotFound()
NotFound: 404: Not Found
2015-08-07 15:18:05,987 ERROR: /js/ui.core.js [in
/home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py:142]
Traceback (most recent call last):
 File
"/home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py",
line 137, in page_not_found
   response = legacy_application(request.environ, g.start_response)
 File
"/home/system/.virtualenvs/invenio/src/invenio/invenio/legacy/wsgi/__init__.py",
line 485, in application
   raise NotFound()
NotFound: 404: Not Found
2015-08-07 15:18:31,388 ERROR: /js/jquery.min.js [in
/home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py:142]
Traceback (most recent call last):
 File
"/home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py",
line 137, in page_not_found
   response = legacy_application(request.environ, g.start_response)
 File
"/home/system/.virtualenvs/invenio/src/invenio/invenio/legacy/wsgi/__init__.py",
line 485, in application
   raise NotFound()
NotFound: 404: Not Found
2015-08-07 15:18:31,444 ERROR: /js/ui.core.js [in
/home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py:142]
Traceback (most recent call last):
 File
"/home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py",
line 137, in page_not_found
   response = legacy_application(request.environ, g.start_response)
 File
"/home/system/.virtualenvs/invenio/src/invenio/invenio/legacy/wsgi/__init__.py",
line 485, in application
   raise NotFound()
NotFound: 404: Not Found
2015-08-07 15:19:18,285 ERROR:  [in
/home/system/.virtualenvs/invenio/src/invenio/invenio/ext/logging/wrappers.py:310]
Traceback (most recent call last):
 File
"/home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py",
line 119, in __call__
   response = self.app.full_dispatch_request()
 File
"/home/system/.virtualenvs/invenio/local/lib/python2.7/site-packages/flask/app.py",
line 1477, in full_dispatch_request
   rv = self.handle_user_exception(e)
 File
"/home/system/.virtualenvs/invenio/local/lib/python2.7/site-packages/flask_restful/__init__.py",
line 270, in error_router
   return original_handler(e)
 File
"/home/system/.virtualenvs/invenio/src/invenio/invenio/base/wrappers.py",
line 133, in handle_user_exception
   return super(Flask, self).handle_user_exception(e)
 File
"/home/system/.virtualenvs/invenio/local/lib/python2.7/site-packages/flask/app.py",
line 1381, in handle_user_exception
   reraise(exc_type, exc_value, tb)
 File
"/home/system/.virtualenvs/invenio/local/lib/python2.7/site-packages/flask/app.py",
line 1475, in full_dispatch_request
   rv = self.dispatch_request()
 File
"/home/system/.virtualenvs/invenio/local/lib/python2.7/site-packages/flask/app.py",
li

Re: TypeError when calling oaiharvest from CLI

2015-08-07 Thread Jan Aage Lavik 
Hi,

I see the problem now. Unfortunately, with the changes in 2.0 not all parts of 
the legacy admin UI was fixed. The in browser display of harvests is one of 
those. Instead, you can use the `oaiharvest` command-line tool to schedule a 
harvest in `bibsched`, our legacy task scheduler.

E.g.  

$ oaiharvest -r desy —dates=-mm-dd:-mm-dd# to avoid harvesting the 
entire repo

Then start bibched:

$ bibsched start

You can monitor the process by accessing bibsched:

$ bibsched

Press ‘h’ for a list of help commands.

To be able to fully run the harvest you also need to make sure a Celery worker 
instance is running as described here:
http://invenio.readthedocs.org/en/latest/getting-started/first-steps.html?highlight=celery#development
 


This will execute a process which will process the harvested data and then 
finally create bibupload jobs that you can run in bibsched (if you tick off 
upload step in post-process).

Word of warning, though, this version of Invenio is a developer preview release 
and not fully stable. We are working hard towards the next release to address 
these and other issues.

Hope this helps. 

Cheers,
Jan

> On 07 Aug 2015, at 15:25, Werner Greßhoff  
> wrote:
> 
> Hello!
> 
> First things first: changing the workflow to 
> 'oaiharvest_harvest_repositories' doesn't change the result!
> So here is now the log:
> 
> 2015-08-07 15:18:05,935 ERROR: /js/jquery.min.js [in 
> /home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py:142]
> Traceback (most recent call last): 
>  File 
> "/home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py",
>  line 137, in page_not_found 
>response = legacy_application(request.environ, g.start_response) 
>  File 
> "/home/system/.virtualenvs/invenio/src/invenio/invenio/legacy/wsgi/__init__.py",
>  line 485, in application 
>raise NotFound() 
> NotFound: 404: Not Found 
> 2015-08-07 15:18:05,987 ERROR: /js/ui.core.js [in 
> /home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py:142]
> Traceback (most recent call last): 
>  File 
> "/home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py",
>  line 137, in page_not_found 
>response = legacy_application(request.environ, g.start_response) 
>  File 
> "/home/system/.virtualenvs/invenio/src/invenio/invenio/legacy/wsgi/__init__.py",
>  line 485, in application 
>raise NotFound() 
> NotFound: 404: Not Found 
> 2015-08-07 15:18:31,388 ERROR: /js/jquery.min.js [in 
> /home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py:142]
> Traceback (most recent call last): 
>  File 
> "/home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py",
>  line 137, in page_not_found 
>response = legacy_application(request.environ, g.start_response) 
>  File 
> "/home/system/.virtualenvs/invenio/src/invenio/invenio/legacy/wsgi/__init__.py",
>  line 485, in application 
>raise NotFound() 
> NotFound: 404: Not Found 
> 2015-08-07 15:18:31,444 ERROR: /js/ui.core.js [in 
> /home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py:142]
> Traceback (most recent call last): 
>  File 
> "/home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py",
>  line 137, in page_not_found 
>response = legacy_application(request.environ, g.start_response) 
>  File 
> "/home/system/.virtualenvs/invenio/src/invenio/invenio/legacy/wsgi/__init__.py",
>  line 485, in application 
>raise NotFound() 
> NotFound: 404: Not Found 
> 2015-08-07 15:19:18,285 ERROR:  [in 
> /home/system/.virtualenvs/invenio/src/invenio/invenio/ext/logging/wrappers.py:310]
> Traceback (most recent call last): 
>  File 
> "/home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py",
>  line 119, in __call__ 
>response = self.app.full_dispatch_request() 
>  File 
> "/home/system/.virtualenvs/invenio/local/lib/python2.7/site-packages/flask/app.py",
>  line 1477, in full_dispatch_request 
>rv = self.handle_user_exception(e) 
>  File 
> "/home/system/.virtualenvs/invenio/local/lib/python2.7/site-packages/flask_restful/__init__.py",
>  line 270, in error_router 
>return original_handler(e) 
>  File 
> "/home/system/.virtualenvs/invenio/src/invenio/invenio/base/wrappers.py", 
> line 133, in handle_user_exception 
>return super(Flask, self).handle_user_exception(e) 
>  File 
> "/home/system/.virtualenvs/invenio/local/lib/python2.7/site-packages/flask/app.py",
>  line 1381, in handle_user_exception 
>reraise(exc_type, exc_value, tb) 
>  File 
> "/home/system/.virtualenvs/invenio/local/lib/python2.7/site-packages/flask/app.py",
>  line 1475, in full_dispatch_request 
>rv = self.dispatch_request() 
>  File 
> "/home/system/.virtualenvs/invenio/local/lib/python2.7/site-packages/flask/app.py",
>  line 1461, in dispatch_request 
>return self.view_

Re: TypeError when calling oaiharvest from CLI

2015-08-07 Thread Werner Greßhoff 

Hello!

First things first: changing the workflow to 
'oaiharvest_harvest_repositories' doesn't change the result!

So here is now the log:

Konsole output
2015-08-07 15:18:05,935 ERROR: /js/jquery.min.js [in 
/home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py:142]

Traceback (most recent call last):
 File 
"/home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py", 
line 137, in page_not_found

   response = legacy_application(request.environ, g.start_response)
 File 
"/home/system/.virtualenvs/invenio/src/invenio/invenio/legacy/wsgi/__init__.py", 
line 485, in application

   raise NotFound()
NotFound: 404: Not Found
2015-08-07 15:18:05,987 ERROR: /js/ui.core.js [in 
/home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py:142]

Traceback (most recent call last):
 File 
"/home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py", 
line 137, in page_not_found

   response = legacy_application(request.environ, g.start_response)
 File 
"/home/system/.virtualenvs/invenio/src/invenio/invenio/legacy/wsgi/__init__.py", 
line 485, in application

   raise NotFound()
NotFound: 404: Not Found
2015-08-07 15:18:31,388 ERROR: /js/jquery.min.js [in 
/home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py:142]

Traceback (most recent call last):
 File 
"/home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py", 
line 137, in page_not_found

   response = legacy_application(request.environ, g.start_response)
 File 
"/home/system/.virtualenvs/invenio/src/invenio/invenio/legacy/wsgi/__init__.py", 
line 485, in application

   raise NotFound()
NotFound: 404: Not Found
2015-08-07 15:18:31,444 ERROR: /js/ui.core.js [in 
/home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py:142]

Traceback (most recent call last):
 File 
"/home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py", 
line 137, in page_not_found

   response = legacy_application(request.environ, g.start_response)
 File 
"/home/system/.virtualenvs/invenio/src/invenio/invenio/legacy/wsgi/__init__.py", 
line 485, in application

   raise NotFound()
NotFound: 404: Not Found
2015-08-07 15:19:18,285 ERROR:  [in 
/home/system/.virtualenvs/invenio/src/invenio/invenio/ext/logging/wrappers.py:310]

Traceback (most recent call last):
 File 
"/home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py", 
line 119, in __call__

   response = self.app.full_dispatch_request()
 File 
"/home/system/.virtualenvs/invenio/local/lib/python2.7/site-packages/flask/app.py", 
line 1477, in full_dispatch_request

   rv = self.handle_user_exception(e)
 File 
"/home/system/.virtualenvs/invenio/local/lib/python2.7/site-packages/flask_restful/__init__.py", 
line 270, in error_router

   return original_handler(e)
 File 
"/home/system/.virtualenvs/invenio/src/invenio/invenio/base/wrappers.py", line 
133, in handle_user_exception

   return super(Flask, self).handle_user_exception(e)
 File 
"/home/system/.virtualenvs/invenio/local/lib/python2.7/site-packages/flask/app.py", 
line 1381, in handle_user_exception

   reraise(exc_type, exc_value, tb)
 File 
"/home/system/.virtualenvs/invenio/local/lib/python2.7/site-packages/flask/app.py", 
line 1475, in full_dispatch_request

   rv = self.dispatch_request()
 File 
"/home/system/.virtualenvs/invenio/local/lib/python2.7/site-packages/flask/app.py", 
line 1461, in dispatch_request

   return self.view_functions[rule.endpoint](**req.view_args)
 File 
"/home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py", 
line 164, in web_admin

   handler=legacy_publisher)
 File 
"/home/system/.virtualenvs/invenio/src/invenio/invenio/legacy/wsgi/__init__.py", 
line 472, in application

   handler(req)
 File 
"/home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py", 
line 162, in legacy_publisher

   possible_handler)
 File 
"/home/system/.virtualenvs/invenio/src/invenio/invenio/legacy/wsgi/__init__.py", 
line 630, in mp_legacy_publisher
   return _check_result(req, module_globals[possible_handler](req, 
**form))

 File "", line 551, in preview_original_xml
 File 
"/home/system/.virtualenvs/invenio/src/invenio/invenio/legacy/oaiharvest/admin.py", 
line 1146, in perform_request_preview_original_xml

   from invenio.legacy.oaiharvest.dblayer import get_oai_src_by_id
ImportError: cannot import name get_oai_src_by_id
2015-08-07 15:19:18,326 ERROR: Exception on 
/admin/oaiharvest/oaiharvestadmin.py/preview_original_xml [GET] [in 
/home/system/.virtualenvs/invenio/local/lib/python2.7/site-packages/flask/app.py:1423]

Traceback (most recent call last):
 File 
"/home/system/.virtualenvs/invenio/src/invenio/invenio/ext/legacy/__init__.py", 
line 119, in __call__

   response = self.app.full_dispatch_request()
 File 
"/home/system/.virtualenvs/invenio/local/lib/python2.7/site-packages/flask/app.py", 
line 1477, in full_dispatch_request

   rv = self

Re: TypeError when calling oaiharvest from CLI

2015-08-07 Thread Jan Åge Lavik 
Hello Werner,

So the Internal Server Error happens when you save the repository in the
OAI Harvest Admin UI? Can you dig our the logs from the server which has
more details? Say, check in cdvirtualenv var/log/invenio.base.log file.

Regarding the workflow to choose, it is a bit confusing right now due to
the state of the migration from legacy to new modules, but you can try
selecting the workflows: 'oaiharvest_harvest_repositories' (which reads
post-process arguments) or 'oaiharvest_repository_with_approval' (which
uses an approval step in the "Holding Pen" available under
/admin/holdingpen).

Cheers,
Jan

---
Jan Age Lavik
System Developer
INSPIRE-HEP 

Github: @jalavik 
Work phone: +41 22 76 78682

On Fri, Aug 7, 2015 at 2:40 PM, Werner Greßhoff <
werner.gressh...@uni-muenster.de> wrote:

> Hi!
>
> I tried the first URL leading to the same Internal Server Error as
> with our own repository. In the OAI Harvest Admin UI I entered a
> name (Desy), the URL (https://bib-pubdb1.desy.de/oai2d) and chose
> marcxml as Metadata-Prefix and the insert Workflow (which I replaced
> with marcxml_workflow after first Internal Server Error). The rest
> of the page (deprecated postprocessing options) I left blank. Then
> I searched for an identifier and tested the harvesting process.
>
> Where am I going wrong?
>
> Best regards,
> Werner
>
> Am 07.08.2015 um 12:28 schrieb Wagner, Alexander:
>
>> Hi!
>>
>> The URL I tried was http://repositorium-dev.uni-muenster.de/oai/miami.
>>> This is our staging system, which is so far the only one to produce
>>> marcxml output.
>>>
>>
>> If it's only about to get Marc records, you may try one of
>> the join2 repos, e.g.
>>
>> https://bib-pubdb1.desy.de/oai2d
>> https://impulse.mlz-garching.de/oai2d
>> https://juser.fz-juelich.de/oai2d
>> https://publications.rwth-aachen.de/oai2d
>> https://repository.gsi.de/oai2d
>>
>> However, I'd suggest to try the openaire or vdb sets. If you
>> harvest blindly you'll get our authority records. They are
>> nice as well, but probably not what you're striving for.
>>
>>
>
> --
> Mit freundlichen Grüßen
>
> Werner Greßhoff
> Dezernat 2 - Digitale Dienste
> Universitäts- und Landesbibliothek Münster
>

Re: TypeError when calling oaiharvest from CLI

2015-08-07 Thread Werner Greßhoff 

Hi!

I tried the first URL leading to the same Internal Server Error as
with our own repository. In the OAI Harvest Admin UI I entered a
name (Desy), the URL (https://bib-pubdb1.desy.de/oai2d) and chose
marcxml as Metadata-Prefix and the insert Workflow (which I replaced
with marcxml_workflow after first Internal Server Error). The rest
of the page (deprecated postprocessing options) I left blank. Then
I searched for an identifier and tested the harvesting process.

Where am I going wrong?

Best regards,
Werner

Am 07.08.2015 um 12:28 schrieb Wagner, Alexander:

Hi!


The URL I tried was http://repositorium-dev.uni-muenster.de/oai/miami.
This is our staging system, which is so far the only one to produce
marcxml output.


If it's only about to get Marc records, you may try one of
the join2 repos, e.g.

https://bib-pubdb1.desy.de/oai2d
https://impulse.mlz-garching.de/oai2d
https://juser.fz-juelich.de/oai2d
https://publications.rwth-aachen.de/oai2d
https://repository.gsi.de/oai2d

However, I'd suggest to try the openaire or vdb sets. If you
harvest blindly you'll get our authority records. They are
nice as well, but probably not what you're striving for.




--
Mit freundlichen Grüßen

Werner Greßhoff
Dezernat 2 - Digitale Dienste
Universitäts- und Landesbibliothek Münster
<>

Re: TypeError when calling oaiharvest from CLI

2015-08-07 Thread Wagner, Alexander 
Hi!

> The URL I tried was http://repositorium-dev.uni-muenster.de/oai/miami.
> This is our staging system, which is so far the only one to produce
> marcxml output.

If it's only about to get Marc records, you may try one of
the join2 repos, e.g.

https://bib-pubdb1.desy.de/oai2d
https://impulse.mlz-garching.de/oai2d
https://juser.fz-juelich.de/oai2d
https://publications.rwth-aachen.de/oai2d
https://repository.gsi.de/oai2d

However, I'd suggest to try the openaire or vdb sets. If you
harvest blindly you'll get our authority records. They are
nice as well, but probably not what you're striving for.

-- 
Kind regards,

Alexander Wagner

Deutsches Elektronen-Synchrotron DESY
Library and Documentation

Building 01d Room OG1.444
Notkestr. 85
22607 Hamburg

phone:  +49-40-8998-1758
fax:+49-40-8994-1758
e-mail: alexander.wag...@desy.de

Re: TypeError when calling oaiharvest from CLI

2015-08-06 Thread Werner Greßhoff 

Dear Jan,

thank you for the patch! I guess it works, but the tested repository
is too slow, so I'm running in a timeout now!
The URL I tried was http://repositorium-dev.uni-muenster.de/oai/miami.
This is our staging system, which is so far the only one to produce
marcxml output. You can't access it, it's only internal!
You can access http://repositorium.uni-muenster.de/oai/miami instead,
but there is only support for oai_dc, mods, mets and epicur.
And yes, I tried to setup the OAI data provider as OAI PMH source.
Initially it's all working, the sets are retrieved...

Best regards,
Werner

Am 06.08.2015 um 12:09 schrieb Jan Åge Lavik:

Dear Werner,

First of all, welcome to Invenio and apologies for the late response.

Indeed you have identified a problem in the legacy oaiharvest tool. I
have provided a fix here:
https://github.com/inveniosoftware/invenio/pull/3428

If you are familiar with git you can try applying this patch using git
cherry-pick inside your virtualenv sources.

Note that this oaiharvest tool is currently being migrated to a separate
package called Invenio-OAIHarvester
(https://github.com/inveniosoftware/invenio-oaiharvester) which will be
available soon. This will contain a new command line tool and web interface.

Regarding the Internal Server Error you experienced, which URL were you
trying to access? Did you try setting up an OAI PMH source in
http://localhost:4000/admin/oaiharvest/oaiharvestadmin.py?

Cheers,
Jan

---
Jan Age Lavik
System Developer
INSPIRE-HEP 

Github: @jalavik 
Work phone: +41 22 76 78682

On Wed, Aug 5, 2015 at 11:01 AM, Werner Greßhoff
mailto:werner.gressh...@uni-muenster.de>> wrote:

Hello,

at first I want to say, I'm new to Invenio and Python, so it might be
my error or a misunderstanding!
We've installed the Invenio 2.1-Version, installation was successful!
Now we are trying to harvest from our existing repository some metadata
beside the metadata delivered with the demo site. The call from the
web frontend led to an "Internal Server Error", so I tried the CLI
instead with the following command:

oaiharvest -vListRecords -pmarcxml -f2004-04-01 -u2004-05-31
-o/tmp/marc.xml http://repositorium-dev.uni-muenster.de/oai/miami

Leading to the following message:

Traceback (most recent call last):
File "./oaiharvest", line 9, in 
  load_entry_point('invenio==2.1.1.dev20150616',
'console_scripts', 'oaiharvest')()
File
"/home/system/.virtualenvs/invenio/src/invenio/invenio/base/helpers.py",
line 50, in decorated_func
  result = f(*args, **kwargs)
File

"/home/system/.virtualenvs/invenio/src/invenio/invenio/legacy/oaiharvest/scripts/oaiharvest.py",
line 51, in main
  return oai_main()
File
"/home/system/.virtualenvs/invenio/src/invenio/invenio/base/helpers.py",
line 52, in decorated_func
  result = f(*args, **kwargs)
File

"/home/system/.virtualenvs/invenio/src/invenio/invenio/legacy/oaiharvest/daemon.py",
line 398, in main
  dummy2, dummy3) = urllib.parse(base_url)
TypeError: 'Module_six_moves_urllib_parse' object is not callable

As a beginner in Python I've no idea what is going wrong. I started
with looking at the output of sys.path, but that's directing to
the invenio library paths, so I guess Python is importing the correct
library.
Does someone have a clue where I'm going wrong?!

--
Mit freundlichen Grüßen

Werner Greßhoff
Dezernat 2 - Digitale Dienste
Universitäts- und Landesbibliothek Münster





--
Mit freundlichen Grüßen

Werner Greßhoff
Dezernat 2 - Digitale Dienste
Universitäts- und Landesbibliothek Münster
<>

Re: TypeError when calling oaiharvest from CLI

2015-08-06 Thread Jan Åge Lavik 
Dear Werner,

First of all, welcome to Invenio and apologies for the late response.

Indeed you have identified a problem in the legacy oaiharvest tool. I have
provided a fix here: https://github.com/inveniosoftware/invenio/pull/3428

If you are familiar with git you can try applying this patch using git
cherry-pick inside your virtualenv sources.

Note that this oaiharvest tool is currently being migrated to a separate
package called Invenio-OAIHarvester (
https://github.com/inveniosoftware/invenio-oaiharvester) which will be
available soon. This will contain a new command line tool and web interface.

Regarding the Internal Server Error you experienced, which URL were you
trying to access? Did you try setting up an OAI PMH source in
http://localhost:4000/admin/oaiharvest/oaiharvestadmin.py?

Cheers,
Jan

---
Jan Age Lavik
System Developer
INSPIRE-HEP 

Github: @jalavik 
Work phone: +41 22 76 78682

On Wed, Aug 5, 2015 at 11:01 AM, Werner Greßhoff <
werner.gressh...@uni-muenster.de> wrote:

> Hello,
>
> at first I want to say, I'm new to Invenio and Python, so it might be
> my error or a misunderstanding!
> We've installed the Invenio 2.1-Version, installation was successful!
> Now we are trying to harvest from our existing repository some metadata
> beside the metadata delivered with the demo site. The call from the
> web frontend led to an "Internal Server Error", so I tried the CLI
> instead with the following command:
>
> oaiharvest -vListRecords -pmarcxml -f2004-04-01 -u2004-05-31
> -o/tmp/marc.xml http://repositorium-dev.uni-muenster.de/oai/miami
>
> Leading to the following message:
>
> Traceback (most recent call last):
>File "./oaiharvest", line 9, in 
>  load_entry_point('invenio==2.1.1.dev20150616', 'console_scripts',
> 'oaiharvest')()
>File
> "/home/system/.virtualenvs/invenio/src/invenio/invenio/base/helpers.py",
> line 50, in decorated_func
>  result = f(*args, **kwargs)
>File
> "/home/system/.virtualenvs/invenio/src/invenio/invenio/legacy/oaiharvest/scripts/oaiharvest.py",
> line 51, in main
>  return oai_main()
>File
> "/home/system/.virtualenvs/invenio/src/invenio/invenio/base/helpers.py",
> line 52, in decorated_func
>  result = f(*args, **kwargs)
>File
> "/home/system/.virtualenvs/invenio/src/invenio/invenio/legacy/oaiharvest/daemon.py",
> line 398, in main
>  dummy2, dummy3) = urllib.parse(base_url)
> TypeError: 'Module_six_moves_urllib_parse' object is not callable
>
> As a beginner in Python I've no idea what is going wrong. I started
> with looking at the output of sys.path, but that's directing to
> the invenio library paths, so I guess Python is importing the correct
> library.
> Does someone have a clue where I'm going wrong?!
>
> --
> Mit freundlichen Grüßen
>
> Werner Greßhoff
> Dezernat 2 - Digitale Dienste
> Universitäts- und Landesbibliothek Münster
>