Sorry if I'm missing something, but I'm not seeing how this fixes the hard coded '_postgres' username in calendarserver_bootstrap_database script. I still can't bootstrap the database using the script because it doesn't pull database login information from the caldavd.plist file, it's hardcoded. What am I missing?
Thanks, Jim On Wed, Dec 28, 2011 at 3:35 PM, Andre LaBranche <d...@apple.com> wrote: > > On Dec 28, 2011, at 11:29 AM, Jim J wrote: > > Thanks Andre! > > The idea works, not so much the implementation though. I'll post what I've > found, perhaps I missed something otherwise it might be something to look > into or at least help others. > > I was able to set up the database schema but I couldn't do it using the > bootstrap Python script. The script imports > calendarserver.tools.bootstrapdatabase and runs main(). The > bootstrapdatabase.py file makes some improper (and un-overrideable) > assumptions. First, CONNECTNAME is assumed to be "_postgres" but on Ubuntu > it's just "postgres" (no leading underscore). Maybe this isn't a bad > assumption on Mac OS X Server? Second it assumes the existence of database > user "caldav" and database "caldav". Unfortunately I'd already committed to > 'caldavd' (trailing 'd') being the username and group for the daemon so > that didn't work either. > > > See the 'database connectivity' section in this document: > > http://trac.calendarserver.org/wiki/MultiServerDeployment > > Basically you just need to set the postgres DSN as appropriate for your > setup. > > Cheers, > -dre > > > I went into the Python file and changed those. Then it assumed that the > SQL script that it needed to execute was located at > /usr/share/caldavd/lib/python/.../current.sql. However, when I installed > calendarserver using ./run -i that's not where it put the SQL script! It > put it in /usr/local/lib/python2.7/dist-packages/.../current.sql. I ended > up quitting at this point and logged into PostgresSQL as the caldavd user > and manually executed the SQL script. > > Finally I was able to get calendarserver fully running *without SSL > encryption*. When I went to enable SSL, I seem to have tripped over some > crazy bug in the twisted framework. I didn't dig too deep but in Twisted > 11.0.0 there's a file internet/tcp.py. In the Connection class there's an > "if SSL:" and then a bunch of functions. Inside _startTLS there's a line: > > self.fileno = self.socket.fileno > > This line asserts. I don't really know why. Something about > OpenSSH.SSL.Connection doesn't have attribute 'fileno'. Anyway, I gave up > at this point, locked the unencrypted calendarserver to localhost and > wrapped it in stunnel4. That seems to work. That line doesn't seem to exist > in Twisted 10.x or even in 11.1.0 so I'm guessing it was a problem. > > Thanks for the help. > > Jim > > On Tue, Dec 27, 2011 at 7:10 PM, Andre LaBranche <d...@apple.com> wrote: > >> Forgot to cc the list... See quoted text for answers :) >> >> -dre >> >> Sent from my iPhone >> >> On Dec 27, 2011, at 6:34 PM, Andre LaBranche <d...@apple.com> wrote: >> >> We only provision schema if we spawn Postgres. To use an externally >> managed DB, provision schema with calendarserver_bootstrap_database. >> >> Cheers, >> -dre >> >> Sent from my iPhone >> >> On Dec 27, 2011, at 4:33 PM, Jim J <jahose...@gmail.com> wrote: >> >> Andre, >> >> Thanks! That's a very useful link actually. I've been trying to figure >> out what all the options are. For some reason it didn't catch my attention >> during my 'grep's. That fixes all the above problems, I can even use -u and >> -g now. >> >> Next problem: >> I created the caldavd user in PostgreSQL and created the caldavd database >> (I'm using -u caldavd -g caldavd again). I noticed this connects to the >> database using the caldavd user. Now I get this error: >> >> 2011-12-27 16:17:47-0500 [-] >> [txdav.common.datastore.util.UpgradeDatabaseSchemaService#warn] Beginning >> database schema check. >> 2011-12-27 16:17:47-0500 [-] >> [txdav.common.datastore.util.UpgradeDatabaseSchemaService#warn] Required >> schema version: 3. >> 2011-12-27 16:17:47-0500 [-] Unhandled error in Deferred: >> 2011-12-27 16:17:47-0500 [-] Unhandled Error >> Traceback (most recent call last): >> File >> "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 388, in >> errback >> self._startRunCallbacks(fail) >> File >> "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 455, in >> _startRunCallbacks >> self._runCallbacks() >> File >> "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 542, in >> _runCallbacks >> current.result = callback(current.result, *args, **kw) >> File >> "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1076, in >> gotResult >> _inlineCallbacks(r, g, deferred) >> --- <exception caught here> --- >> File >> "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1018, in >> _inlineCallbacks >> result = result.throwExceptionIntoGenerator(g) >> File >> "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 350, in >> throwExceptionIntoGenerator >> return g.throw(self.type, self.value, self.tb) >> File >> "/usr/local/lib/python2.7/dist-packages/txdav/common/datastore/util.py", >> line 271, in doUpgrade >> actual_version = yield sqlTxn.schemaVersion() >> File >> "/usr/lib/python2.7/dist-packages/twisted/internet/defer.py", line 1018, in >> _inlineCallbacks >> result = result.throwExceptionIntoGenerator(g) >> File >> "/usr/lib/python2.7/dist-packages/twisted/python/failure.py", line 350, in >> throwExceptionIntoGenerator >> return g.throw(self.type, self.value, self.tb) >> File >> "/usr/local/lib/python2.7/dist-packages/txdav/common/datastore/sql.py", >> line 233, in schemaVersion >> result = yield self._schemaVersion.on(self) >> File >> "/usr/local/lib/python2.7/dist-packages/twext/internet/threadutils.py", >> line 64, in _oneWorkUnit >> result = instruction() >> File >> "/usr/local/lib/python2.7/dist-packages/twext/enterprise/adbapi2.py", line >> 155, in <lambda> >> lambda : self._reallyExecSQL(*args, **kw) >> File >> "/usr/local/lib/python2.7/dist-packages/twext/enterprise/adbapi2.py", line >> 136, in _reallyExecSQL >> result = self._reallyExecSQL(sql, args, raiseOnZeroRowCount) >> File >> "/usr/local/lib/python2.7/dist-packages/twext/enterprise/adbapi2.py", line >> 130, in _reallyExecSQL >> self._cursor.execute(sql, args) >> File >> "/usr/local/lib/python2.7/dist-packages/txdav/base/datastore/dbapiclient.py", >> line 76, in execute >> self.realCursor.execute(sql, args) >> File "/usr/local/lib/python2.7/dist-packages/pgdb.py", line >> 259, in execute >> self.executemany(operation, (params,)) >> File "/usr/local/lib/python2.7/dist-packages/pgdb.py", line >> 289, in executemany >> raise DatabaseError("error '%s' in '%s'" % (msg, sql)) >> pg.DatabaseError: error 'ERROR: relation "calendarserver" does >> not exist >> LINE 1: select VALUE from CALENDARSERVER where NAME = 'VERSION' >> ^ >> ' in 'select VALUE from CALENDARSERVER where NAME = 'VERSION'' >> >> >> I would have expected calendarserver to set up its own tables/schema (as >> it appears to be attempting to do). Can I force it to rebuild the database? >> >> Thanks, >> >> Jim >> >> On Tue, Dec 27, 2011 at 1:40 PM, Andre LaBranche <d...@apple.com> wrote: >> >>> Hi, >>> >>> Try setting UseDatabase to Postgres, as described here: >>> http://trac.calendarserver.org/browser/CalendarServer/tags/release/CalendarServer-3.1/twistedcaldav/stdconfig.py >>> >>> You will also need to specify DSN which tells calendar server how to >>> connect to postgres. >>> >>> You can also set ServerEnabled to false for memcached to keep calendar >>> server from spawning memcached. >>> >>> I can provide examples once I get back home to a real machine... >>> >>> Cheers, >>> -dre >>> >>> Sent from my iPhone >>> >>> On Dec 27, 2011, at 1:21 PM, Jim J <jahose...@gmail.com> wrote: >>> >>> All, >>> >>> I'm quite new to this calendarserver thing. I'm trying to set up 3.1 >>> (from tags/releases in Subversion) on a 64-bit Ubuntu 11.10 server >>> installation. It's been a disaster. I was able to run calendarserver from >>> the SVN directory using ./run and then I "installed" it using ./run -i. >>> That's where things went bad. Some issues I've worked around, but I'd like >>> to get good solutions for. Currently I'm stuck trying to figure out what's >>> going on with PostgreSQL. Perhaps you guys can clue me in on how to >>> properly fix the following issues. >>> >>> This is how I'm currently executing caldavd. Once it works I'll put it >>> into init.d and run it as a proper server. The () are because I *was* >>> running with those flags (see part 1) and have given up on them. >>> >>> sudo caldavd -X -L -f <path to caldavd.plist file> (-u caldavd -g >>> caldavd) >>> >>> 1. As I understand it, it is good practice to run Linux services that >>> are exposed to the internet as a restricted user. Given that and existence >>> of the -u and -g flags for caldavd, I figured those flags would be the >>> right way to do things. That doesn't work at all. >>> >>> The first problem is that caldavd wants to write to >>> /var/run/ccs_postgresql_xxxxx which it can't do because /var belongs to >>> root and it attempts to do this after the setuid() call to become the >>> caldavd user. So, I went into caldavd.plist and changed the "DataRoot" to a >>> different directory and gave it to caldavd:caldavd. >>> >>> Then I tried again. This time it failed because it appears that >>> calendarserver attempts to start postgresql whether or not the postgresql >>> service is already running on the local box. To do this, it tries to >>> setuid() again, this time to ROOT! If the entire point of setting a user >>> for a service is to restrict it's privileges, why would one of those privs >>> be to allow setuid to root? That makes no sense. >>> >>> So I gave up, now I'm trying to run it as root to see if the 3.1 release >>> even functions. How do I make this server run as somebody other than root? >>> >>> 2. Running as root (without the -u and -g flags shown above) gets the >>> application started and then it fails attempting to start memcached. I get >>> the error "cannot listen on tcp port". This is because memcachd is a system >>> service and is *already running*. Again, why is calendarserver attempting >>> to start a service that's already running? >>> >>> So I stopped memcachd and tried again. >>> >>> 3. Calendarserver now successfully starts and connects to memcached but >>> it cannot connect to postgresql. This I haven't yet worked around. I get >>> the following error: >>> 2011-12-27 12:43:11-0500 [-] [txdav.base.datastore.subpostgres#error] >>> [Failure instance: Traceback: <class 'pg.InternalError'>: could not connect >>> to server: No such file or directory >>> Is the server running locally and accepting >>> connections on Unix domain socket >>> "/tmp/ccs_postgres_f5e263275ce7de576bd42ca08db4fc56/.s.PGSQL.5432"? >>> >>> >>> /usr/lib/python2.7/dist-packages/twisted/internet/_baseprocess.py:60:maybeCallProcessEnded >>> >>> /usr/local/lib/python2.7/dist-packages/txdav/base/datastore/subpostgres.py:92:processEnded >>> >>> /usr/lib/python2.7/dist-packages/twisted/internet/defer.py:361:callback >>> >>> /usr/lib/python2.7/dist-packages/twisted/internet/defer.py:455:_startRunCallbacks >>> --- <exception caught here> --- >>> >>> /usr/lib/python2.7/dist-packages/twisted/internet/defer.py:542:_runCallbacks >>> >>> /usr/local/lib/python2.7/dist-packages/txdav/base/datastore/subpostgres.py:367:gotReady >>> >>> /usr/local/lib/python2.7/dist-packages/txdav/base/datastore/subpostgres.py:269:ready >>> >>> /usr/local/lib/python2.7/dist-packages/txdav/base/datastore/subpostgres.py:261:produceConnection >>> >>> /usr/local/lib/python2.7/dist-packages/txdav/base/datastore/dbapiclient.py:196:connect >>> /usr/local/lib/python2.7/dist-packages/pgdb.py:482:connect >>> ] >>> >>> Here I'm stuck. Quite clearly calendarserver is expecting to launch the >>> postgresql service (which, again, doesn't make sense to me), but then it >>> can't connect when it does. Well, at least I'm assuming calendarserver >>> successfully launches postgresql because I don't get the setuid() error I >>> got in part 1 when it tries to setuid() to root in order to launch >>> postgresql. I can't figure out how to fix this. Any thoughts? >>> >>> Thanks for the help, >>> >>> Jim >>> >>> _______________________________________________ >>> calendarserver-users mailing list >>> calendarserver-users@lists.macosforge.org >>> http://lists.macosforge.org/mailman/listinfo.cgi/calendarserver-users >>> >>> >> >> _______________________________________________ >> calendarserver-users mailing list >> calendarserver-users@lists.macosforge.org >> http://lists.macosforge.org/mailman/listinfo.cgi/calendarserver-users >> >> > >
_______________________________________________ calendarserver-users mailing list calendarserver-users@lists.macosforge.org http://lists.macosforge.org/mailman/listinfo.cgi/calendarserver-users
