Hi,
I have attached the ds_backup.py code. Tack on all disclaimers - it has
not been tested in the XS-0.7 and 12.1.0 environment. It will need some
revision to deal with the 'shared' XO environment in Lesotho so I will
work on it while I am there.
However, it really is not changed from XS-0.7 in areas that matter to
you. The changes are in the way the Journal objects are handled. Martin
Langhoff used rsync so that the backup is a snapshot of the local
datastore. I am trying to establish the server-side as the real Journal
with the local datastore containing only the currently relevant content.
It seems sad that kids must delete their Journal because of the limited
size of the local store. The message: Journal is full is not helpful,
because the fact is that the store is full, often because of the
excessive number of installed activities. With a school server, this is
totally unnecessary since the Sugar Activities are available on the
school server so that unused ones can be deleted on the XO.
I had to move to mod_wsgi for Django (library). As far as I know, there
is no change needed client-side. Apache needs to have wsgi installed (in
XS-0.7 it is). Then all that is needed is a file in httpd/conf.d.
I have also attached the one I am using.
On 04/07/2013 06:04 AM, George Hunt wrote:
Hi Tony,
I'm playing with ds-backup now, because one of the dependencies for
ds-backup is mod_python which was dropped from fedora 18, in favor of
mod_wsgi. It seems like a great time to rethink the issue.
I cannot find the working code you mention in this email in my google
stack. Can you send it to me again?
I'm also talking with Tim Moody, about using puppet client as a way of
pushing and modifying the configuration of XO's in the classroom. Do
you have any experience, or ideas, about such a proposal? I'm not sure
whether ds-backup might be used to introduce puppet client back into XOs.
George
On Sun, Dec 9, 2012 at 12:25 PM, Tony Anderson <[email protected]
<mailto:[email protected]>> wrote:
Hi,
I will send you the scripts as soon as I have a chance to try them
on 12.0.1 and XS-0.7. Last year they were running on xs-0.6 and
build 852 at Saint Jacob. There shouldn't be too much difference,
although so far on most of the code I have been wrong by about 3 days!
The identity problem really isn't that hard. A lot of the mailing
list discussion seems related to running Sugar on something besides
an XO.
The user is identified on the server by the serial-number. For the
backup, this is not a problem. The problem comes from <OLPC where an
XO is shared. Then there needs to be something like a login to
identify the actual user. There also needs to be separate Journals
on the XO and separate backups on the school server.
I should be able to give you working code within a week (I leave
Kigali on 12/23 so it must be before then).
Yours,
Tony
On 12/09/2012 04:11 PM, George Hunt wrote:
Wow Tony,
Your solution definitely needs to be part of the mix as we go
forward.
Can you send me copies of the scripts, or the changes you made to
accomplish these objectives? If we are going to look at the
serialnumber-user-identity issue, we might be making changes to
the same
packages.
Have you worked through the changes needed to add user identity to
journal backups? As Paul Fox was suggesting, I think there will
be lots
of ripple effects, if and when we start adding additional users, in
addition to the user "olpc". Or maybe, someone will come up with a
simplifying assumption, or approach.
George
On Sun, Dec 9, 2012 at 4:49 AM, Tony Anderson
<[email protected] <mailto:[email protected]>
<mailto:[email protected] <mailto:[email protected]>>__>
wrote:
Hi, Sameer
I got an email from Nick Doiron re his visit to the Marshall
Islands. He mentioned that you gave him a script to create
a csv
from the Journal backup.
As I have mentioned several times on the list, I believe
Martin's
backup scheme while elegantly implemented is not adequate.
His model
is the traditional backup/restore.
The problem, as always, is storage space. When an XO-1 is
out of
space, the user gets a message 'Journal is full'. The only
practical
solution often is to reflash the XO losing all of the Journal
objects. If a user deletes an object from the Journal on
the XO, the
rsynch also removes it from the backup.
I modified the scripts ds-backup.sh and ds-backup.py on the
XO to
use a different paradigm. The registration process creates two
scripts in the /library/users/serial-number folder on the
school
server: journal and log. The backup script uploads the
object to
journal if there is an associated data file; otherwise, it is
uploaded to the log folder and deleted from the local
store. The
remaining objects are marked as favorites in the Journal
Activity to
show that the data file is available on the XO. If the user
clears
the star, the data file is deleted from the datastore but
remains on
the school server. If the user fills a clear star, the
backup script
downloads the data file from the school server.
At Saint Jacob, all of the laptops are 4gb XO-1.5 so I have
not had
to implement storage management. The plan would be to set
limits on
the size of the datastore and the /home/olpc/Activities
folders. If
a datastore outgrows the limit, data files can be deleted LRU.
Similarly, activities can be deleted LRU. This way the user
will not
have to be involved. The user can always request a needed
file to be
restored to the local datastore and can request an activity be
downloaded and installed from the school server repository.
I expect
this will be needed in Lesotho for XO-1.
This method also supports restoring the users Journal after
a reflash.
It also enables a replacement laptop by renaming the
serial-number
folder on the school server (and updating the registration
information).
I am not sure if you are trying to implement this
functionality, but
it is important at a deployment.
Yours,
Tony
LoadModule wsgi_module modules/mod_wsgi.so
#!/usr/bin/python
# -*- coding: utf-8 -*-
#updated version
# Copyright (C) 2007 Ivan KrstiÄ
# Copyright (C) 2007 Tomeu Vizoso
# Copyright (C) 2007 One Laptop per Child
#
# This program is free software; you can redistribute it and/or modify
# it under the terms of version 2 of the GNU General Public License (and
# no other version) as published by the Free Software Foundation.
#
# This program is distributed in the hope that it will be useful,
# but WITHOUT ANY WARRANTY; without even the implied warranty of
# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
# GNU General Public License for more details.
#
# You should have received a copy of the GNU General Public License
# along with this program; if not, write to the Free Software
# Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
import os, re, sys, tempfile, glob, time
import urllib2
from urllib2 import URLError, HTTPError, urlopen
import subprocess
from subprocess import Popen, PIPE, call
from sugar import env
from sugar import profile
from sugar.datastore import datastore
from path import path
from sftp import sftp
import datetime
LIMIT = 1024
WORKPATH = path('/tmp/')
EXCLUDED = ['org.laptop.Terminal', 'org.laptop.Log']
class BackupError(Exception): pass
class ProtocolVersionError(BackupError): pass
class RefusedByServerError(BackupError): pass
class ServerTooBusyError(BackupError): pass
class TransferError(BackupError): pass
class NoPriorBackups(BackupError): pass
class BulkRestoreUnavailable(BackupError): pass
def check_server_available(server, xo_serial):
try:
ret = urllib2.urlopen(server + '/available/%s' % xo_serial).read()
return 200
except HTTPError, e:
# server is there, did not fullfull req
# expect 404, 403, 503 as e[1]
return e.code
except URLError, e:
# log it?
# print e.reason
return -1
def sync_time_to_xs():
xstime = urlopen('http://schoolserver/bibliotheque/school/time')
response = xstime.read().replace('\n','')
txtout = 'sudo date --set="' + response + '"'
print txtout
fout = open('/tmp/xstime','w')
fout.write(txtout)
fout.close()
def datastore_to_xs():
#open log
log = open('/tmp/logBackup','w')
#get journal records
query = {}
keys = ['uid','activity','title','title_set_by_user','mime_type','journal','keep']
ds_objects,num_objects = datastore.find(query,properties=keys)
print >> log, 'num_objects',num_objects
#process each one
for i in range(num_objects):
ds_object = ds_objects[i]
#is it in the journal
try:
journal = ds_object.metadata['journal']
except:
print >> log,sys.exc_info()[:2]
#new record created since last run of ds-backup
#is it a journal record? (title_set_by_user == 1 and has_file_path)
print >> log, i, 'new record'
activity = ds_object.metadata['activity']
if activity in EXCLUDED:
jr = False
print >> log, 'activity',activity,'excluded'
else:
jr = True
title_set_by_user = ds_object.metadata['title_set_by_user']
file_path = ds_object.get_file_path()
jobject = datastore.get(ds_object.object_id)
metadata = {}
for key in jobject.metadata.keys():
if not key == 'preview':
metadata[key] = jobject.metadata[key]
metadata_path = WORKPATH+path(file_path).namebase+'.metadata'
fout = open(metadata_path,'w')
fout.write(str(metadata))
fout.close()
if path(file_path).exists() and title_set_by_user == '1' and jr:
#journal record
print >>log, 'journal record',file_path
try:
ds_object.metadata['journal'] = 1
except:
print >> log, 'create journal key failed',sys.exc_info()[:2]
ds_object.metadata['keep'] = 1
script = 'cd journal\nput '+ metadata_path+'\nput '+file_path+'\n'
result,err = sftp(script,WORKPATH)
if err:
print >> log, script, 'err', err
print >> log, ds_object.metadata['title'],'added to Journal',
print >> log, 'keep', ds_object.metadata['keep']
datastore.write(ds_object)
else:
#log record
print >> log, 'log record'
script = 'cd log\nput '+metadata_path+'\n'
result,err = sftp(script,WORKPATH)
if err:
print >> log, script, 'err', err
else:
try:
datastore.delete(ds_object.object_id)
except:
pass
subprocess.call('rm -rf ' + WORKPATH / '*.metadata',shell=True)
else:
#journal record
file_path = ds_object.get_file_path()
fn = path(file_path).name
try:
testjournal = ds_object.metadata['journal']
except:
testjournal = 'key error'
try:
testkeep = ds_object.metadata['keep']
except:
testkeep = 'key error'
print >> log, i, 'existing journal record', fn,
print >> log, 'journal',testjournal,
print >> log, 'keep',testkeep
if journal == 2:
print >> log, 'delete requested'
#delete it from ss
file_path = ds_object.get_filepath
fn = path(file_path).name
script = 'cd journal\nrm -rf ' + fn + '*'
result,err = sftp(script)
if result:
print >> log, 'result',script,result
if err:
print >> log, 'err',script,err
#delete locally
try:
datastore.delete(ds_object.object_id)
except:
pass
else:
try:
keep = ds_object.metadata['keep']
except:
keep = 'key error'
try:
journal = ds_object.metadata['journal']
except:
journal = 'key error'
print >> log, 'journal record: keep', keep, 'journal', journal
if keep == 0 and journal == 1:
#delete local data
print >> log, 'deleting local data'
dsobject.file_path = None
dsobject.metadata['journal'] = 0
datastore.write(ds_object)
if keep == 1 and journal == 0:
#get local copy of data from ss
print >> log, 'getting local copy of data from ss'
script = 'cd journal\nget '+fn
result,err=sftp(script,WORKPATH)
if result:
print >> log, 'result', script, result
if err:
print >> log, 'err',script,err
dsobject.file_path = path('/tmp') / fn
dsobject.metadata['journal'] = 1
datastore.write(ds_object)
#log version
pth = path('/home/olpc/.sugar/default/patch')
if pth.exists():
fin = open(pth,'r')
version = fin.read()
fin.close()
metadata = {}
metadata['mtime'] = datetime.date
metadata['title'] = 'version'
metadata['title_set_by_user'] = 1
metadata['version'] = version
metadata_path = WORKPATH+path(file_path).namebase+'.metadata'
fout = open(metadata_path,'w')
fout.write(str(metadata))
fout.close()
script = 'cd log\nput '+metadata_path+'\n'
result,err = sftp(script,WORKPATH)
if err:
print >> log, script, 'err', err
else:
try:
datastore.delete(ds_object.object_id)
except:
pass
log.close()
def get_sn():
if have_ofw_tree():
return read_ofw('mfg-data/SN')
# on SoaS try gconf, 'identifiers'
sn = gconf_get_string('/desktop/sugar/soas_serial')
if sn:
return sn
sn = identifier_get_string('sn')
if sn:
return sn
return 'SHF00000000'
def get_backup_url():
bu = gconf_get_string('/desktop/sugar/backup_url')
if bu:
return bu
try: # pre-gconf
from iniparse import INIConfig
conf = INIConfig(open(os.path.expanduser('~')+'/.sugar/default/config'))
# this access mode throws an exception if the value
# does not exist
bu = conf['Server']['backup1']
except:
pass
if bu:
return bu
bu = identifier_get_string('backup_url')
if bu:
return bu
return ''
def gconf_get_string(key):
"""We cannot use python gconf from cron scripts,
but cli gconftool-2 does the trick.
Will throw subprocess.Popen exceptions"""
try:
value = Popen(['gconftool-2', '-g', key],
stdout=PIPE).communicate()[0]
return value
except:
return ''
def identifier_get_string(key):
"""This is a config method used by some versions of
Sugar -- in use in some SoaS"""
try:
fpath = os.path.expanduser('~')+'/.sugar/default/identifiers/'+key
fh = open(fpath, 'r')
value = fh.read().rstrip('\0\n')
fh.close()
return value
except:
return ''
def have_ofw_tree():
return os.path.exists('/ofw')
def read_ofw(path):
path = os.path.join('/ofw', path)
if not os.path.exists(path):
return None
fh = open(path, 'r')
data = fh.read().rstrip('\0\n')
fh.close()
return data
_______________________________________________
Server-devel mailing list
[email protected]
http://lists.laptop.org/listinfo/server-devel
