I was able to work around the issues as follows
For the upgrade issue - my workaround:
diff /root/invenio_2012_11_15_bibdocfile_model.py
/opt/invenio/lib/python/invenio/upgrades/invenio_2012_11_15_bibdocfile_model.py
151d150
< continue
167c166
< new_value = '' #cPickle.dumps(minfo['flags'])
---
> new_value = cPickle.dumps(minfo['flags'])
196,206c195,205
< # try:
< # flags = cPickle.loads(run_sql("SELECT data_value
FROM bibdocmoreinfo WHERE id_bibdoc=%s AND namespace=%s AND
data_key=%s", (str(docid), '', 'flags'))[0][0])
< # for flagname in minfo['flags']:
< # for version in minfo['flags'][flagname]:
< # for docformat in
minfo['flags'][flagname][version]:
< # if
minfo['flags'][flagname][version][docformat]:
< # are_equal = are_equal and
(docformat in flags[flagname][version])
< # if not (docformat in
flags[flagname][version]):
< # logger.info("ERROR: Document
%s: Expected %s" % (str(docid), str(minfo) ))
< # except Exception, e:
< # logger.info("ERROR: Document %s: Problem with
retrieving flags. %s MoreInfo: %s flags: %s" % (str(docid), str(e),
str(minfo), str(flags)))
---
> try:
> flags = cPickle.loads(run_sql("SELECT data_value
FROM bibdocmoreinfo WHERE id_bibdoc=%s AND namespace=%s AND
data_key=%s", (str(docid), '', 'flags'))[0][0])
> for flagname in minfo['flags']:
> for version in minfo['flags'][flagname]:
> for docformat in
minfo['flags'][flagname][version]:
> if
minfo['flags'][flagname][version][docformat]:
> are_equal = are_equal and (docformat
in flags[flagname][version])
> if not (docformat in
flags[flagname][version]):
> logger.info("ERROR: Document %s:
Expected %s" % (str(docid), str(minfo) ))
> except Exception, e:
> logger.info("ERROR: Document %s: Problem with
retrieving flags. %s MoreInfo: %s flags: %s" % (str(docid), str(e),
str(minfo), str(flags)))
And for the cache issue - the collection name had a '/' in name. I
renamed with dash as a workaround:
update collection set name="DOE-NSF Reviews" where id=42;
------------------------------------------------------------------------
*From: *gary holman
*Posted: *17/03/2015 18:29
*Subject: *Invenio-1.2.0 upgrade issues from 99.3
Hi All,
I have been having a few issues upgrading our cds-invenio build 99.3
to the lastest stable release invenio-1.2.0. I can successfully update
to version 1.1.5 without error. Upgrading from 1.1.5 to 1.2.0 also fails.
I am building the new invenio using the maint-1.2 git branch on Debian
Wheezy (SMP Debian 3.2.65-1+deb7u2 x86_64 GNU/Linux)
I list 2 upgrade issues below. I can bypass this update (deleting
invenio_2012_11_15_bibdocfile_model.py) and the upgrade completes as
expected. However I then experience another issue with webcoll (also
described below).
Any ideas?
The two problems updating to 1.2.0 are as follows:
---------------------------------
-------------
1. When upgrading 'sudo -u www-data /opt/invenio/bin/inveniocfg
--upgrade'
-------------
I get errors on the invenio_2012_11_15_bibdocfile_model.py update
Description - my database bibdoc table ids are not contiguous. There
are missing records (the first one happens to be id 1). I can bypass
this issue by adding 'continue' on line 151 in upgrade file
(invenio_2012_11_15_bibdocfile_model.py)
Error:
---------
>>> Applying invenio_2012_11_15_bibdocfile_model (Change of the
underlying data model allowing extended BibDocs and MoreInfo)
>>> invenio_2012_11_15_bibdocfile_model: droping old backup tables
>>> invenio_2012_11_15_bibdocfile_model: Adding missing columns to tables
>>> invenio_2012_11_15_bibdocfile_model: ERROR: Error when migrating
document 1 attached to the record 1: can not retrieve from the bibdoc
table
Traceback (most recent call last):
File "/opt/invenio/bin/inveniocfg", line 33, in <module>
main()
File "/usr/local/lib/python2.7/dist-packages/invenio/inveniocfg.py",
line 1763, in main
cli_cmd_upgrade(conf)
File "/usr/local/lib/python2.7/dist-packages/invenio/inveniocfg.py",
line 1477, in cli_cmd_upgrade
cmd_upgrade(conf)
File
"/usr/local/lib/python2.7/dist-packages/invenio/inveniocfg_upgrader.py",
line 907, in cmd_upgrade
upgrader.apply_upgrade(u)
File
"/usr/local/lib/python2.7/dist-packages/invenio/inveniocfg_upgrader.py",
line 493, in apply_upgrade
upgrade['do_upgrade']()
File
"/opt/invenio/lib/python/invenio/upgrades/invenio_2012_11_15_bibdocfile_model.py",
line 39, in do_upgrade
if not _fix_recid(recid, logger):
File
"/opt/invenio/lib/python/invenio/upgrades/invenio_2012_11_15_bibdocfile_model.py",
line 157, in _fix_recid
if res2[0][1]:
IndexError: tuple index out of range
-------------
2. invenio_2012_11_15_bibdocfile_model.py update
-------------
If I bypass the issue described above, I receive more_info errors
Description - my database bibdoc table bibdoc.moreinfo does not have
'flags' only comments and description
Output from sql ("select id,docname, more_info from bibdoc where
docname='MJProposal07v0'":
---------
id docname more_info
329 MJProposal07v0
(dp1\nS'descriptions'\np2\n(dp3\nsS'comments'\np4\n(dp5\ns.
Is my database/tables corrupt?
Error:
---------
>>> invenio_2012_11_15_bibdocfile_model: Upgrading record with docname
MJProposal07v0:
Traceback (most recent call last):
File "/opt/invenio/bin/inveniocfg", line 33, in <module>
main()
File "/usr/local/lib/python2.7/dist-packages/invenio/inveniocfg.py",
line 1763, in main
cli_cmd_upgrade(conf)
File "/usr/local/lib/python2.7/dist-packages/invenio/inveniocfg.py",
line 1477, in cli_cmd_upgrade
cmd_upgrade(conf)
File
"/usr/local/lib/python2.7/dist-packages/invenio/inveniocfg_upgrader.py",
line 907, in cmd_upgrade
upgrader.apply_upgrade(u)
File
"/usr/local/lib/python2.7/dist-packages/invenio/inveniocfg_upgrader.py",
line 493, in apply_upgrade
upgrade['do_upgrade']()
File
"/opt/invenio/lib/python/invenio/upgrades/invenio_2012_11_15_bibdocfile_model.py",
line 39, in do_upgrade
if not _fix_recid(recid, logger):
File
"/opt/invenio/lib/python/invenio/upgrades/invenio_2012_11_15_bibdocfile_model.py",
line 168, in _fix_recid
new_value = cPickle.dumps(minfo['flags'])
KeyError: 'flags'
The issue after updating 1.2.0 is as follows:
---------------------------------
-------------
3. webcoll error (webcoll crash)? always happens at the exact same
place in webcoll webcache part 2:
-------------
Command:
sudo -u www-data /opt/invenio/bin/webcoll -f -v9
2015-03-13 12:01:43 --> argv=['/opt/invenio/bin/webcoll', '-f', '-v9']
2015-03-13 12:01:43 --> _OPTIONS={'force': 1}
2015-03-13 12:01:43 --> _TASK_PARAMS={'profile': [],
'stop_queue_on_error': False, 'post-process': [],
'task_stop_helper_fnc': None, 'verbose': 9, 'sleeptime': '',
'fixed_time': False, 'runtime_limit': None, 'priority': 0,
'task_specific_name': '', 'version': '$Id$', 'sequence-id': None,
'user': '', 'task_name': 'webcoll', 'host': '', 'runtime': '2015-03-13
12:01:43'}
Results:
file: bibsched_task_244730.log
...
2015-03-13 11:56:56 --> Entering task_sleep_now_if_required with
status=RUNNING
2015-03-13 11:56:56 --> August 2006 Meeting, Seattle, WA / webpage
cache update
2015-03-13 11:56:56 --> ... creating
/opt/invenio/var/cache/collections/August 2006 Meeting, Seattle,
WA-ln=en.html
2015-03-13 11:56:56 --> Updating task progress to Part 2/2: done 15/72.
2015-03-13 11:56:56 --> Entering task_sleep_now_if_required with
status=RUNNING
2015-03-13 11:56:56 --> DOE/NSF Reviews / webpage cache update
2015-03-13 11:56:56 --> Task #244730 finished. [RUNNING]
file: bibsched_task_244730.err is empty
Thank you,
Gary Holman
University of Washington
CENPA
