Package: calendarserver
Followup-For: Bug #722198
It appeared that regression inroduced in sqlparse v 0.1.5 caused exactly this
problem that was recently fixed in 0.1.9 release of the library. The upstream
bug for a previous issue is:
https://github.com/andialbrecht/sqlparse/issues/111
However the upload won't fix the original problem server not being able to
start due to another problem of similar origin. It appears later while parsing
the same current.sql file. The application throws an exception while parsing
the following query:
create table ADDRESSBOOK_BIND (
ADDRESSBOOK_HOME_RESOURCE_ID integer not null references integer
not null references ADDRESSBOOK on delete cascade,
ADDRESSBOOK_RESOURCE_NAME integer not null, --
enum CALENDAR_BIND_MODE
BIND_STATUS integer not null, -- enum
CALENDAR_BIND_STATUS
SEEN_BY_OWNER boolean not null,
SEEN_BY_SHAREE boolean not null,
MESSAGE text, -- FIXME: xml?
primary key(ADDRESSBOOK_HOME_RESOURCE_ID, ADDRESSBOOK_RESOURCE_ID), --
implicit index
unique(ADDRESSBOOK_HOME_RESOURCE_ID, ADDRESSBOOK_RESOURCE_NAME) --
implicit index
);
and exactly while parsing "--" string (of ttype "Token.Operator") in the
following line:
ADDRESSBOOK_RESOURCE_NAME integer not null, --
enum CALENDAR_BIND_MODE
I've attached full traceback at the end of my message. I've also done some
simple traceback for difference between sql tokens parsed by sqlparse 0.1.4 and
0.1.9 as well as a whole sql file and a 5-lines script to check results. I
believe the problem is the difference that causes this error is between line 42
of 0.1.4.txt file and line 37 of 0.1.9.txt (you may use any diff program like
vimdiff to get a good visualization of changes). It is possible that previous
difference (in cascade statements differently parsed by 0.1.4 and 0.1.9) is
somehow related as well.
Currently waiting for an answer from upstream developer.
Detailed trace of my attempt running calendarserver with sqlparse upgraded to
0.1.9:
Traceback (most recent call last):
File "/usr/bin/twistd", line 14, in <module>
run()
File "/usr/lib/python2.7/dist-
packages/twisted/scripts/twistd.py", line 27, in run
app.run(runApp, ServerOptions)
File "/usr/lib/python2.7/dist-packages/twisted/application/app.py", line 647,
in run
config.parseOptions()
File "/usr/lib/python2.7/dist-packages/twisted/application/app.py", line 614,
in parseOptions
usage.Options.parseOptions(self, options)
File "/usr/lib/python2.7/dist-packages/twisted/python/usage.py", line 261, in
parseOptions
for (cmd, short, parser, doc) in self.subCommands:
File "/usr/lib/python2.7/dist-packages/twisted/application/app.py", line 631,
in subCommands
for plug in sorted(plugins, key=attrgetter('tapname')):
File "/usr/lib/python2.7/dist-packages/twisted/plugins/caldav.py", line 29,
in getProperty
return getattr(reflect.namedClass(self.serviceMakerClass), propname)
File "/usr/lib/python2.7/dist-packages/twisted/python/_reflectpy3.py", line
151, in namedObject
module = namedModule('.'.join(classSplit[:-1]))
File "/usr/lib/python2.7/dist-packages/twisted/python/_reflectpy3.py", line
137, in namedModule
topLevel = __import__(name)
File "/usr/lib/python2.7/dist-packages/twistedcaldav/mail.py", line 78, in
<module>
from calendarserver.tap.util import getRootResource, directoryFromConfig
File "/usr/lib/python2.7/dist-packages/calendarserver/tap/util.py", line 89,
in <module>
from txdav.common.datastore.sql import CommonDataStore as CommonSQLDataStore
File "/usr/lib/python2.7/dist-packages/txdav/common/datastore/sql.py", line
47, in <module>
from txdav.common.datastore.sql_legacy import
PostgresLegacyNotificationsEmulator
File "/usr/lib/python2.7/dist-packages/txdav/common/datastore/sql_legacy.py",
line 51, in <module>
from txdav.common.datastore.sql_tables import (
File "/usr/lib/python2.7/dist-packages/txdav/common/datastore/sql_tables.py",
line 44, in <module>
schema = _populateSchema()
File "/usr/lib/python2.7/dist-packages/txdav/common/datastore/sql_tables.py",
line 40, in _populateSchema
return SchemaSyntax(schemaFromPath(pathObj))
File "/usr/lib/python2.7/dist-packages/twext/enterprise/dal/parseschema.py",
line 86, in schemaFromPath
addSQLToSchema(schema, schemaData)
File "/usr/lib/python2.7/dist-packages/twext/enterprise/dal/parseschema.py",
line 115, in addSQLToSchema
t = tableFromCreateStatement(schema, stmt)
File "/usr/lib/python2.7/dist-packages/twext/enterprise/dal/parseschema.py",
line 70, in tableFromCreateStatement
cp.parse()
File "/usr/lib/python2.7/dist-packages/twext/enterprise/dal/parseschema.py",
line 222, in parse
while self.nextColumn():
File "/usr/lib/python2.7/dist-packages/twext/enterprise/dal/parseschema.py",
line 236, in nextColumn
return self.parseConstraint(maybeIdent)
File "/usr/lib/python2.7/dist-packages/twext/enterprise/dal/parseschema.py",
line 267, in parseConstraint
raise ViolatedExpectation('PRIMARY or UNIQUE', constraintType)
twext.enterprise.dal.parseschema.ViolatedExpectation: Expected 'PRIMARY or
UNIQUE' got --
Debug scripts and results on my host attached.
-- System Information:
Debian Release: jessie/sid
APT prefers unstable
APT policy: (990, 'unstable'), (500, 'stable'), (500, 'oldstable'), (1,
'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386
Kernel: Linux 3.2.0-4-amd64 (SMP w/4 CPU cores)
Locale: LANG=uk_UA.UTF-8, LC_CTYPE=uk_UA.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Versions of packages calendarserver depends on:
ii adduser 3.113+nmu3
ii libc6 2.17-93
ii lsb-base 4.1+Debian12
ii memcached 1.4.13-0.2
ii python 2.7.5-5
ii python-dateutil 1.5+dfsg-0.1
ii python-kerberos 1.1+svn4895-1+b2
ii python-openssl 0.13-2.1
ii python-plist 1.10-1
ii python-pycalendar 2.0~svn188-1
ii python-pygresql 1:4.0-3
ii python-pysqlite2 2.6.3-3
ii python-sqlparse 0.1.4-1
ii python-twisted-conch 1:13.0.0-1
ii python-twisted-core 13.0.0-1
ii python-twisted-mail 13.0.0-1
ii python-twisted-web 13.0.0-1
ii python-twisted-words 13.0.0-1
ii python-xattr 0.6.4-2
ii python-zope.interface 4.0.5-1+b1
ii ssl-cert 1.0.33
Versions of packages calendarserver recommends:
pn python-ldap <none>
pn python-pam <none>
calendarserver suggests no packages.
-- Configuration Files:
/etc/default/calendarserver changed [not included]
-- no debconf information
sqlparse.__version__ is 0.1.4
Token.Punctuation u'('
Token.Text.Whitespace.Newline u'\n'
Token.Text.Whitespace u' '
None u'ADDRESSBOOK_HOME_RESOURCE_ID'
Token.Text.Whitespace u' '
Token.Name.Builtin u'integer'
Token.Text.Whitespace u' '
Token.Keyword u'not null'
Token.Text.Whitespace u' '
Token.Keyword u'references'
Token.Text.Whitespace u' '
None u'ADDRESSBOOK_HOME,\n ADDRESSBOOK_RESOURCE_ID'
Token.Text.Whitespace u' '
Token.Name.Builtin u'integer'
Token.Text.Whitespace u' '
Token.Keyword u'not null'
Token.Text.Whitespace u' '
Token.Keyword u'references'
Token.Text.Whitespace u' '
None u'ADDRESSBOOK'
Token.Text.Whitespace u' '
Token.Keyword u'on'
Token.Text.Whitespace u' '
Token.Keyword.DML u'delete'
Token.Text.Whitespace u' '
Token.Keyword u'cascade'
Token.Punctuation u','
Token.Text.Whitespace.Newline u'\n'
Token.Text.Whitespace.Newline u'\n'
Token.Text.Whitespace u' '
None u"-- An invitation which hasn't been accepted yet will not yet have a
resource\n -- name, so this field may be null.\n\n "
None u'ADDRESSBOOK_RESOURCE_NAME'
Token.Text.Whitespace u' '
None u'varchar(255),\n BIND_MODE'
Token.Text.Whitespace u' '
Token.Name.Builtin u'integer'
Token.Text.Whitespace u' '
Token.Keyword u'not null'
Token.Punctuation u','
Token.Text.Whitespace u' '
None u'-- enum CALENDAR_BIND_MODE\n '
None u'BIND_STATUS'
Token.Text.Whitespace u' '
Token.Name.Builtin u'integer'
Token.Text.Whitespace u' '
Token.Keyword u'not null'
Token.Punctuation u','
Token.Text.Whitespace u' '
None u'-- enum CALENDAR_BIND_STATUS\n '
None u'SEEN_BY_OWNER'
Token.Text.Whitespace u' '
Token.Name.Builtin u'boolean'
Token.Text.Whitespace u' '
Token.Keyword u'not null'
Token.Punctuation u','
Token.Text.Whitespace.Newline u'\n'
Token.Text.Whitespace u' '
None u'SEEN_BY_SHAREE'
Token.Text.Whitespace u' '
Token.Name.Builtin u'boolean'
Token.Text.Whitespace u' '
Token.Keyword u'not null'
Token.Punctuation u','
Token.Text.Whitespace.Newline u'\n'
Token.Text.Whitespace u' '
None u'MESSAGE'
Token.Text.Whitespace u' '
Token.Name.Builtin u'text'
Token.Punctuation u','
Token.Text.Whitespace u' '
None u'-- FIXME: xml?\n\n '
Token.Keyword u'primary'
Token.Text.Whitespace u' '
Token.Keyword u'key'
None u'(ADDRESSBOOK_HOME_RESOURCE_ID, ADDRESSBOOK_RESOURCE_ID)'
Token.Punctuation u','
Token.Text.Whitespace u' '
None u'-- implicit index\n '
Token.Keyword u'unique'
None u'(ADDRESSBOOK_HOME_RESOURCE_ID, ADDRESSBOOK_RESOURCE_NAME)'
Token.Text.Whitespace u' '
None u'-- implicit index\n'
Token.Punctuation u')'
sqlparse.__version__ is 0.1.9
Token.Punctuation u'('
Token.Text.Whitespace.Newline u'\n'
Token.Text.Whitespace u' '
None u'ADDRESSBOOK_HOME_RESOURCE_ID'
Token.Text.Whitespace u' '
Token.Name.Builtin u'integer'
Token.Text.Whitespace u' '
Token.Keyword u'not null'
Token.Text.Whitespace u' '
Token.Keyword u'references'
Token.Text.Whitespace u' '
None u'ADDRESSBOOK_HOME,\n ADDRESSBOOK_RESOURCE_ID'
Token.Text.Whitespace u' '
Token.Name.Builtin u'integer'
Token.Text.Whitespace u' '
Token.Keyword u'not null'
Token.Text.Whitespace u' '
Token.Keyword u'references'
Token.Text.Whitespace u' '
None u'ADDRESSBOOK'
Token.Text.Whitespace u' '
Token.Keyword u'on'
Token.Text.Whitespace u' '
Token.Keyword.DML u'delete'
Token.Text.Whitespace u' '
None u"cascade,\n\n -- An invitation which hasn't been accepted yet will
not yet have a resource\n -- name, so this field may be null.\n\n "
None u'ADDRESSBOOK_RESOURCE_NAME'
Token.Text.Whitespace u' '
None u'varchar(255),\n BIND_MODE'
Token.Text.Whitespace u' '
Token.Name.Builtin u'integer'
Token.Text.Whitespace u' '
Token.Keyword u'not null'
Token.Punctuation u','
Token.Text.Whitespace u' '
Token.Operator u'--'
Token.Text.Whitespace u' '
None u'enum'
Token.Text.Whitespace u' '
None u'CALENDAR_BIND_MODE'
Token.Text.Whitespace.Newline u'\n'
Token.Text.Whitespace u' '
None u'BIND_STATUS'
Token.Text.Whitespace u' '
Token.Name.Builtin u'integer'
Token.Text.Whitespace u' '
None u'not null, -- enum CALENDAR_BIND_STATUS\n '
None u'SEEN_BY_OWNER'
Token.Text.Whitespace u' '
Token.Name.Builtin u'boolean'
Token.Text.Whitespace u' '
None u'not null,\n SEEN_BY_SHAREE'
Token.Text.Whitespace u' '
Token.Name.Builtin u'boolean'
Token.Text.Whitespace u' '
None u'not null,\n MESSAGE'
Token.Text.Whitespace u' '
Token.Name.Builtin u'text'
Token.Punctuation u','
Token.Text.Whitespace u' '
None u'-- FIXME: xml?\n\n '
Token.Keyword u'primary'
Token.Text.Whitespace u' '
Token.Keyword u'key'
None u'(ADDRESSBOOK_HOME_RESOURCE_ID, ADDRESSBOOK_RESOURCE_ID)'
Token.Punctuation u','
Token.Text.Whitespace u' '
None u'-- implicit index\n '
Token.Keyword u'unique'
None u'(ADDRESSBOOK_HOME_RESOURCE_ID, ADDRESSBOOK_RESOURCE_NAME)'
Token.Text.Whitespace u' '
None u'-- implicit index\n'
Token.Punctuation u')'
#!/usr/bin/env python
import sqlparse
soup = sqlparse.parse(open("current.sql").read())
y = create_table_addressbook_bin_args = soup[56].tokens[-2].tokens[-1]
print "sqlparse.__version__ is {0}".format(sqlparse.__version__)
print "\n".join(["{0}\t{1!r}".format(x.ttype, x.value) for x in y.tokens])
