Hi Ludmila,
I've made the following:
$ chown -R apache.apache /opt/invenio/
Restarted bibencode daemon as apache user.
Resubmitted demo-video.
Then .
>This means that there is a permission error somewhere. The building of
BibDoc object from what is on disk has failed due to >some permissions
problem when trying to write the .md5 file. I have sent a patch for
BibDocFile to raise an error instead of >silently just registering the
exception, so it will be more obvious in the future when this problem
occurs. (Which version of >Invenio are you running, so I can sent you the
patch?)
Invenio v.1.1.1
>Regarding why it is happening and how to solve it, it's a bit more tricky
to answer without access to the machine, since it's not >an Invenio issue.
>You can try to resubmit the video and check a few things, like:
> - bibencode process is started as apache user
>- on disk, in the bibdoc folder, the the previous case the 255/ folder,
what are the access rights for all files, including the .md5 ($ > ls -la) -
check if the apache user has rights to write.
Now all files, including .md5 are of apache owner.
But the browser does not display webm-files.
I checked if they are in BibDocFile. They are not there.
Then I built them manually:
$ bibdoc._build_file_list()
They are built, but still not displayed by browser.
Any idea?
Genis
On 06/28/2013 11:35 AM, Genis Musulmanbekov wrote:
Hi Ludmila,
Indeed, 2 webm-files are created in BibDocFile:
[BibDocFile('/opt/invenio/var/data/files/g0/255/Demo_video.mpg;master;1',
'Main', 1, 'Demo_video', '.mpg;master', 6300, 255, '',
'23a0260d8d40eeb8e29e069902f9e9ec', BibDocMoreInfo(255,
"(dp1\nS'flags'\np2\n(dp3\nsS'descriptions'\np4\n(dp5\nI1\n(dp6\nS'.mpg?subf
ormat=master'\np7\nS'MASTER'\np8\nsS'.mpg;master'\np9\nS'MASTER'\np10\nsS'.w
ebm;480p'\np11\nS'WEBM_480P'\np12\nsS'.webm;720p'\np13\nS'WEBM_720P'\np14\ns
ssS'comments'\np15\n(dp16\nI1\n(dp17\nS'.mpg?subformat=master'\np18\nS'MASTE
R'\np19\nsS'.webm;480p'\np20\nS'WEBM_480P'\np21\nsS'.mpg;master'\np22\nS'MAS
TER'\np23\nsS'.webm;720p'\np24\nS'WEBM_720P'\np25\nsss."), False),
BibDocFile('/opt/invenio/var/data/files/g0/255/Demo_video.webm;480p;1',
'Main', 1, 'Demo_video', '.webm;480p', 6300, 255, '',
'66d8b8a168c205077853de09539edaed', BibDocMoreInfo(255,
"(dp1\nS'flags'\np2\n(dp3\nsS'descriptions'\np4\n(dp5\nI1\n(dp6\nS'.mpg?subf
ormat=master'\np7\nS'MASTER'\np8\nsS'.mpg;master'\np9\nS'MASTER'\np10\nsS'.w
ebm;480p'\np11\nS'WEBM_480P'\np12\nsS'.webm;720p'\np13\nS'WEBM_720P'\np14\ns
ssS'comments'\np15\n(dp16\nI1\n(dp17\nS'.mpg?subformat=master'\np18\nS'MASTE
R'\np19\nsS'.webm;480p'\np20\nS'WEBM_480P'\np21\nsS'.mpg;master'\np22\nS'MAS
TER'\np23\nsS'.webm;720p'\np24\nS'WEBM_720P'\np25\nsss."), False),
BibDocFile('/opt/invenio/var/data/files/g0/255/Demo_video.webm;720p;1',
'Main', 1, 'Demo_video', '.webm;720p', 6300, 255, '',
'c96fb805bdecb0d7425b7b12d9ce0ede', BibDocMoreInfo(255,
"(dp1\nS'flags'\np2\n(dp3\nsS'descriptions'\np4\n(dp5\nI1\n(dp6\nS'.mpg?subf
ormat=master'\np7\nS'MASTER'\np8\nsS'.mpg;master'\np9\nS'MASTER'\np10\nsS'.w
ebm;480p'\np11\nS'WEBM_480P'\np12\nsS'.webm;720p'\np13\nS'WEBM_720P'\np14\ns
ssS'comments'\np15\n(dp16\nI1\n(dp17\nS'.mpg?subformat=master'\np18\nS'MASTE
R'\np19\nsS'.webm;480p'\np20\nS'WEBM_480P'\np21\nsS'.mpg;master'\np22\nS'MAS
TER'\np23\nsS'.webm;720p'\np24\nS'WEBM_720P'\np25\nsss."), False)]
What to do further?
Genis
From: Ludmila Marian [mailto:[email protected]]
Sent: Thursday, June 27, 2013 9:12 PM
To: Genis Musulmanbekov
Cc: Jerome Caffaro; Jean-Yves Le Meur; project-invenio-general
Subject: Re: FW: video upload
Hi Genis,
Not sure if you still have problems with this..
If yes, can you also try to see what BibDocFiles are attached to the video
bibdoc:
$ from invenio.bibdocfile import BibRecDocs
$ brd = BibRecDocs(6259)
$ bibdoc = brd.get_bibdoc('Demo_video')
$ bibdoc.list_all_files() # this should return a list of 3 BibDocFiles, 1
for master and 2 for webm slaves
If the above only returns the master, which looks like the problem you are
having, this might be due to some restriction issues when trying to update
the bibdoc structure, more exactly when trying to update the .md5 file. This
can happen if the user process that runs the bibencode jobs does not have
the necessary permissions to access the .md5 file in write mode.
One thing to do in this case is to check the permissions of the .md5 file in
the 243 dir. They should be the same for all files, allowing the 'www-data'
user (or your apache user) to write.
Another thing that you can try in order to figure out if this is the problem
(since indeed the logs are not very helpful in this case) is to open ipython
as superuser (so you know you have all the rights), and try to redo the list
of bibdocfile attached to the Demo_video bibdoc:
$ from invenio.bibdocfile import *
$ brd = BibRecDocs(6259)
$ bibdoc = brd.get_bibdoc('Demo_video')
$ bibdoc.list_all_files() # this should show only the master
$ bibdoc._build_file_list() # this should reconstruct the list of
bibdocfiles attached to this bibdoc
$ bibdoc.list_all_files() # should now return a list of 3 BibDocFiles,
including the webm slaves, in which case the display should be ok
This is a simplified manner of reproducing what bibencode is doing. If this
works, it means that there is a mis-configuration somewhere regarding
permissions. The good thing is that since the the webm slaves are in the
bibdoc folder it means that the conversion worked :-)
Cheers,
Ludmila
On 06/07/2013 05:36 PM, Genis Musulmanbekov wrote:
Hi Ludmila,
>Oh, you're probably not running latest master. In this case this should
work:
>brd.get_bibdoc('Demo_video').get_base_dir()
The response:
***
<bound method BibDoc.get_base_dir of BibDoc(243, 6295, 'Demo_video', 'Main',
False)>
***
The contents of dir '243' are:
Demo_video.mpg;master;1
Demo_video.webm;480p;1
Demo_video.webm;720p;1
So, why webm-files are not added to the record?
>In any case, this will only show you what it's in the folder on disk (this
folder should contain the master and webm formats), it >will not solve the
issue. You should check the logs (bibtask and invenio) to see if something
went wrong when trying to add the >webm formats to the record after they
were created.
There are no err-logs except for 'invenio.err' (which I sent you before).
But I can't make clear the source of the error from its content. If it is
not sufficiently informative to solve the problem, is it?
Genis
Cheers,
Ludmila
On 06/07/2013 03:37 PM, Genis Musulmanbekov wrote:
Hi Ludmila,
>Hi Genis,
>You were saying in your previous email that the webm formats were created,
but looking at this record, http://lt-
<http://lt-%3ejds.jinr.ru/record/6295> >jds.jinr.ru/record/6295 , they have
not been added to the record (they're not in the list of files or in the
marc), this is why you >can not see them on the record page. Maybe some
troubles with bibdocfile?
>You could also check the folder where the formats for this record should be
stored to see if they are there.
>For finding out the location on disk:
>from invenio.bibdocfile import BibRecDocs
>brd = BibRecDocs(6259)
>brd.list_bibdocs_by_names()['Demo_video'].get_base_dir()
Yes, I run these commands and the response was
*******
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
AttributeError: BibRecDocs instance has no attribute 'list_bibdocs_by_names'
*******
Genis
Cheers,
Ludmila
On 06/05/2013 08:23 PM, Genis Musulmanbekov wrote:
Hi Ludmila,
>These errors are raised when trying to interpret and display the
bfe_video_platform_suggestions format element (used >for displaying detailed
video pages). Unfortunately the error log is not detailed enough to point to
the error,
But the traceback with detailed info was in the attached 'invenio.err' in my
previous letter. I am sending it again (attached).
>but you can try to debug it using ipython:
$ipython
In [1]: from invenio.bibformat_elements.bfe_video_platform_suggestions
import format_element
In [2]: from invenio.bibformat_engine import BibFormatObject
In [3]: bfo = BibFormatObject(recid) # recid is the record id of your video
record
In [4]: format_element(bfo)
This last call will probably fail, and give you the traceback with
information on where the problem is.
The last call
In [4]: format_element(bfo)
did not fail, and the response for it is
******
'\n<!-- VIDEO SUGGESTION -->\n<div class="video_suggestion_box">\n
<div class="video_suggestion_thumbnail">\n
<a href="http://lt-jds.jinr.ru/record/6295";>\n
<img
src="http://lt-jds.jinr.ru/record/6295/files/Demo_video_06.jpg?subformat=big
" alt="POSTER"/>\n
</a>\n
<div class="video_suggestion_duration">\n 11 s\n </div>\n </div>\n
<div class="video_suggestion_title">\n Demo video: CMS Detector with Heavy
Ion Collision \n </div>\n
<div class="video_suggestion_author">\n by JDS Administrator\n
</div>\n</div>\n'
******
Any idea?
Genis
Hi Genis,
These errors are raised when trying to interpret and display the
bfe_video_platform_suggestions format element (used for displaying detailed
video pages). Unfortunately the error log is not detailed enough to point to
the error, but you can try to debug it using ipython:
$ipython
In [1]: from invenio.bibformat_elements.bfe_video_platform_suggestions
import format_element
In [2]: from invenio.bibformat_engine import BibFormatObject
In [3]: bfo = BibFormatObject(recid) # recid is the record id of your video
record
In [4]: format_element(bfo)
This last call will probably fail, and give you the traceback with
information on where the problem is.
Hope this helps!
Cheers,
Ludmila
On 06/04/2013 01:55 PM, Genis Musulmanbekov wrote:
Dear Jerome,
> I am assuming that in your case the "assure_quality" option is
> preventing the 'mp4' and 'webm' options to be created. Indeed, the
> profiles used in the default "batch_template_submission.json" create
> videos sized at minimum 854x480 (*_480P) while your input videos are
> 720x576 (PAL) max. In order to validate this assumption you could try
> uploading a larger video on your server, such as the demo one (Full
> HD):
> < <http://invenio-demo.cern.ch/record/107/files/CERN-MOVIE-2010-075.mpg>
http://invenio-demo.cern.ch/record/107/files/CERN-MOVIE-2010-075.mpg>
>
Indeed, higher quality video-files (1240x720) uploading resulted in
conversion to webm-formats (480 and 720). However, they are not seen by the
browser and can't be played. Using VIDEO SUBMISSION I uploaded demo-video
'CERN-MOVIE-2010-075.mpg (with another name). webm-files are created but
the result is the same.
There are some errors given by 'invenio.err' (attached) starting with:
"InvenioBibFormatError: "Error when evaluating format element
VIDEO_PLATFORM_SUGGESTIONS with parameters {'threshold': '75', 'bfo':
<invenio.bibformat_engine.BibFormatObject instance at 0x7f9fdc0e44d0>,
'shuffle': 'True', 'maximum': '3', 'collection': 'Videos'}."
(bibformat_engine.py:622:eval_format_element)"
What is the source of them?
Genis
