Spage has uploaded a new change for review.
https://gerrit.wikimedia.org/r/188287
Change subject: Use export_trello logic and trello_PHIds mapping
......................................................................
Use export_trello logic and trello_PHIds mapping
* Refactor to use a TrelloImporter object.
* Identify card members, start trying to map them.
* Change argument to phab_project to match export_trello.py.
* Don't die on no checklist (from normal board export), nor on missing
column in get_column_name.
* Handle idLabels, add get_labelnames command
* Cards in enterprise export have an idLabels array, not a labels array.
* Use card's idList to set the card's column.
* Adapt to use trello_makePHIDs.py output file.
* Handle card shortUrl and shortLink; they're not present in Trello's
Enterprise Export JSON :( , derive from card.url if necessary.
* Use export_trello.py's card logic in trello_import.py.
Import import TrelloCard, TrelloDAO, TrelloScrubber from
export_trello.py. Load the json file into TrelloDAO object,
and use TrelloCard and TrelloScrubber to fill in each card.
* Convert export_trello.py's checklist handling from adding a comment into
adding strings to description. Also fake out export_trello.py's
setup_logging().
Sample run:
* export all Trello boards
* modify /etc/phabtools.conf for either phab-01.wmflabs.org or
phabricator.wmflabs.org
* set up conf/trello-scrub.yaml for names on phab-01 or phabricator
* run trello_makePHIDs.py
* run
python trello_create.py -v -vv -t -d \
-j
trabulous_dir/wikimediafoundation_20150112_213549/boards/flow_backlog.json \
-l 'Send to Phabricator - Collaboration-Team board'
Change-Id: I70897def2a781191fe06213cf157284f4e29b1a6
---
A export_trello.py
M trello_create.py
2 files changed, 861 insertions(+), 70 deletions(-)
git pull ssh://gerrit.wikimedia.org:29418/phabricator/tools
refs/changes/87/188287/1
diff --git a/export_trello.py b/export_trello.py
new file mode 100644
index 0000000..526b5a4
--- /dev/null
+++ b/export_trello.py
@@ -0,0 +1,589 @@
+#!/usr/bin/env python
+
+# This is an example script as part of a trac/trello --> phabricator
+# migration. It may or may not have worked at a particular point in
+# time but could be totally broken by the time you read this. Even if
+# it did work it is still completely tied to the internal
+# idiosyncrasies of a decade of trac use and migration goals of a
+# single company. There will never be any documentation.
+#
+# YOU SHOULD NOT RUN THIS SCRIPT.
+#
+# If you are willing to take COMPLETE RESPONSIBILITY FOR WRECKING YOUR
+# PHABRICATOR INSTALL you should MAYBE consider this script as an
+# EXAMPLE to help your own design. NO ONE CAN HELP YOU. If you were
+# not already willing to write such scripts from scratch run away from
+# the dragon now. It is provided for that purpose and as a
+# demonstration for upstream of a migration to guide feature
+# development that may make that very slightly less painful one day.
+#
+# YOU SHOULD NOT RUN THIS SCRIPT. A DRAGON WILL EAT ALL OF YOUR DATA.
+#
+# /===-_---~~~~~~~~~------____
+# |===-~___ _,-'
+# -==\\ `//~\\ ~~~~`---.___.-~~
+# ______-==| | | \\ _-~`
+# __--~~~ ,-/-==\\ | | `\ ,'
+# _-~ /' | \\ / / \ /
+# .' / | \\ /' / \ /'
+# / ____ / | \`\.__/-~~ ~ \ _ _/' / \/'
+# /-'~ ~~~~~---__ | ~-/~ ( ) /' _--~`
+# \_| / _) ; ), __--~~
+# '~~--_/ _-~/- / \ '-~ \
+# {\__--_/} / \\_>- )<__\ \
+# /' (_/ _-~ | |__>--<__| |
+# |0 0 _/) )-~ | |__>--<__| |
+# / /~ ,_/ / /__>---<__/ |
+# o o _// /-~_>---<__-~ /
+# (^(~ /~_>---<__- _-~
+# ,/| /__>--<__/ _-~
+# ,//('( |__>--<__| / .----_
+# ( ( ')) |__>--<__| | /' _---_~\
+# `-)) )) ( |__>--<__| | /' / ~\`\
+# ,/,'//( ( \__>--<__\ \ /' // ||
+# ,( ( ((, )) ~-__>--<_~-_ ~--____---~' _/'/ /'
+# `~/ )` ) ,/| ~-_~>--<_/-__ __-~ _/
+# ._-~//( )/ )) ` ~~-'_/_/ /~~~~~~~__--~
+# ;'( ')/ ,)( ~~~~~~~~~~
+# ' ') '( (/
+# ' ' `
+
+# This script uses a trello enterprise export which even if you have
+# an enterprise account is different from what you get by hitting the
+# export button on a board.
+
+import argparse
+import calendar
+import collections
+import errno
+import json
+import logging
+import logging.handlers
+import os
+import pprint
+import sys
+import time
+import traceback
+
+import dateutil.parser
+import yaml
+
+##### Utility #####
+
+def mkdir_p(path):
+ try:
+ os.makedirs(path)
+ except OSError as exc: # Python >2.5
+ if exc.errno == errno.EEXIST and os.path.isdir(path):
+ pass
+ else:
+ raise
+
+##### Trello's world view #####
+
+# Why doesn't trello just just unix time and why is python's date/time
+# handling such a mess?
+def parse_trello_ts_str(s):
+ d = dateutil.parser.parse(s)
+ return calendar.timegm(d.utctimetuple())
+
+
+def s_to_us(ts):
+ return ts * 1000000
+
+class TrelloDAO(object):
+
+ # Trello exports a board as a blob of json. It's a werid hybrid
+ # between 'just make a giant json blob' and 'this looks
+ # suspicously like internal represenation'
+ def __init__(self, fname):
+ with open(fname) as f:
+ self.blob = json.load(f)
+ self.uid2label = None
+ self.uid2username = None
+ self.column_id2name = None
+
+ def get_board_id(self):
+ return self.blob['id']
+
+ def get_board_name(self):
+ return self.blob['name']
+
+ def get_board_url(self):
+ return self.blob['url']
+
+ # Label definitions. Like members, an array of dicts
+ def get_labelnames(self):
+ labelnames = []
+ for label in self.blob['labels']:
+ # TODO? the label['idBoard'] might not be the current board, but
if it's unique maybe write it anyway.
+ labelnames.append('%s (%s)' % (label['name'],label['color']))
+ return sorted(labelnames)
+
+ # Returns a string for the Trello label.
+ def get_label(self, uid):
+ if self.uid2label is None:
+ self.uid2label = {}
+ for label in self.blob['labels']:
+ self.uid2label[label['id']] = ('%s (%s)' %
(label['name'],label['color']))
+ if uid in self.uid2label:
+ return self.uid2label[uid]
+ else:
+ return 'UNKNOWN_LABEL_' + uid
+
+
+ def get_usernames(self):
+ usernames = []
+ for user in self.blob['members']:
+ usernames.append(user['username'])
+ return sorted(usernames)
+
+ def get_username(self, uid):
+ if self.uid2username is None:
+ self.uid2username = {}
+ for user in self.blob['members']:
+ self.uid2username[user['id']] = user['username']
+ if uid in self.uid2username:
+ return self.uid2username[uid]
+ else:
+ return 'UNKNOWN_' + uid
+
+ def get_column_name(self, column_id):
+ if self.column_id2name is None:
+ self.column_id2name = {}
+ for t_list in self.blob['lists']:
+ self.column_id2name[t_list['id']] = t_list['name']
+ # This failed in card 108 of a simple Trello export, referencing a
+ # "Done" column that is no longer part of this board, or maybe the card
+ # was on a different board.
+ try:
+ column_name = self.column_id2name[column_id]
+ except KeyError as e:
+ log.warn('get_column_name found no name for column id' + column_id)
+ column_name = 'NOTFOUND-'+column_id
+ self.column_id2name[column_id] = column_name
+ return column_name
+
+
+ # due to cards moving and whatnot constraints like 'there should
+ # be a created record for each card' can not be satisfied
+ def figure_out_when_card_created(self, card):
+ eldest_record = parse_trello_ts_str(card.dateLastActivity)
+ has_create_record = False
+ for action in self.blob['actions']:
+ if action['type'] == 'createCard':
+ if action['data']['card']['id'] == card.card_id:
+ change_time = parse_trello_ts_str(action['date'])
+ if change_time < eldest_record:
+ eldest_record = change_time
+ has_create_record = True
+ if not has_create_record:
+ for action in self.blob['actions']:
+ if action['type'] == 'updateCard':
+ if action['data']['card']['id'] == card.card_id:
+ change_time = parse_trello_ts_str(action['date'])
+ if change_time < eldest_record:
+ eldest_record = change_time
+ return eldest_record
+
+ def figure_out_first_card_column(self, card):
+ for action in self.blob['actions']:
+ if action['type'] == 'createCard' and action['data']['card']['id']
== card.card_id:
+ if 'list' in action['data']:
+ try:
+ return
self.get_column_name(action['data']['list']['id'])
+ except KeyError as e:
+ pass
+ # column IDs can be referenced that are not defined
+ return 'UNKNOWN'
+
+
+ def guess_card_reporter(self, card, scrubber):
+ reporter = None
+ for action in self.blob['actions']:
+ if action['type'] == 'createCard':
+ if action['data']['card']['id'] == card.card_id:
+ reporter =
scrubber.get_phab_uid(self.get_username(action['idMemberCreator']))
+ # TODO: spagewmf: found a reporter, break out of the loop
+ if reporter is None and card.idMembers:
+ reporter =
scrubber.get_phab_uid(self.get_username(card.idMembers[0]))
+ return reporter if reporter else 'import-john-doe'
+
+
+ def guess_card_owner(self, card, scrubber):
+ owner = None
+ if card.idMembers:
+ owner = scrubber.get_phab_uid(self.get_username(card.idMembers[0]))
+ return owner if owner else 'import-john-doe'
+
+ def figure_out_all_subscribers(self, card, scrubber):
+ subscriber_ids = set(card.idMembers)
+ for action in self.blob['actions']:
+ if 'card' in action['data'] and action['data']['card']['id'] ==
card.card_id:
+ subscriber_ids.add(action['idMemberCreator'])
+ subscribers = map(lambda s:
scrubber.get_phab_uid(self.get_username(s)), subscriber_ids)
+ return sorted(subscribers)
+
+
+ # Note: this method is unused.
+ def get_checklist_items(self, card_id):
+ for c_list in self.blob['checklists']: # TODO Only in Enterprise
export, not available in board export
+ if c_list['idCard'] == card_id:
+ check_items = c_list['checkItems']
+ return sorted(check_items, key=lambda e: e['pos'])
+
+
+ def get_relevant_actions(self, card_id):
+ actions = []
+ for action in self.blob['actions']:
+ if 'card' in action['data'] and action['data']['card']['id'] ==
card_id:
+ if action['type'] in ['commentCard', 'updateCard']:
+ actions.append(action)
+ return sorted(actions, key=lambda a: parse_trello_ts_str(a['date']))
+
+class TrelloScrubber(object):
+
+ def __init__(self, conf_file):
+ with open(conf_file) as f:
+ self.conf = yaml.load(f)
+
+
+ def get_phab_uid(self, trello_username):
+ # trello exports can include user ids that are are not defined
+ # as members or anywhere within in the export
+ if trello_username.startswith('UNKNOWN_'):
+ junk = trello_username.split('UNKNOWN_')[1]
+ if junk in self.conf['uid-cheatsheet']:
+ return self.conf['uid-cheatsheet'][junk]
+ else:
+ return 'FAILED-'+trello_username # TODO log error
+ else:
+ if trello_username in self.conf['uid-map']:
+ return self.conf['uid-map'][trello_username]
+ else:
+ return 'FAILED-'+trello_username # TODO log error
+
+
+class TrelloCard(object):
+
+ # [u'attachments', Never used on board
+ # OLD u'labels', Rarely used color stuff
+ # u'idLabels', array of ids of labels for card
+ # u'pos', Physical position, ridiculous LOE to port so ignoring
+ # u'manualCoverAttachment', Duno but it's always false
+ # u'id', unique id
+ # u'badges', something about fogbugz integration?
+ # u'idBoard', parent board id
+ # u'idShort', "short" and thus not unique uid
+ # u'due', rarely used duedate
+ # u'shortUrl', pre-shorted url
+ # u'closed', boolean for if it's archived
+ # u'subscribed', boolean, no idea what it means
+ # u'email', no idea, always none
+ # u'dateLastActivity', 2014-04-22T14:09:49.917Z
+ # u'idList', it's the ID of the current column of the card
+ # u'idMembersVoted', never used
+ # u'idMembers', # Whose face shows up next to it
+ # u'checkItemStates', Something to do with checklists?
+ # u'desc', # description field
+ # u'descData', Almost always None, probably not important
+ # u'name', # title
+ # u'shortLink', # some short linky thing
+ # u'idAttachmentCover', Always None
+ # u'url', # link back to trello
+ # u'idChecklists'] # a bunch of ids for checklists?
+
+ def __init__(self, blob, scrubber):
+ self.scrubber = scrubber
+ self.card_id = blob['id']
+ if "labels" in blob:
+ self.labels = blob['labels']
+ else:
+ self.labels = []
+ self.idLabels = blob['idLabels']
+ self.idBoard = blob['idBoard']
+ self.due = blob['due']
+ self.closed = blob['closed']
+ self.dateLastActivity = blob['dateLastActivity']
+ self.idList = blob['idList']
+ self.idMembers = blob['idMembers']
+ self.desc = blob['desc']
+ self.name = blob['name']
+ self.url = blob['url']
+ # Board export has shortUrl and shortLink, but Enterprise export
doesn't - crazy.
+ if 'shortUrl' in blob:
+ self.shortUrl = blob['shortUrl']
+ else:
+ # Trim the end off e.g.
https://trello.com/c/mpFNXCXp/464-long-title-here
+ self.shortUrl = self.url[0:self.url.rfind('/')]
+ if 'shortLink' in blob:
+ self.shortLink = blob['shortLink']
+ else:
+ # Again, the last piece.
+ self.shortLink = self.shortUrl[self.shortUrl.rfind('/')+1:]
+
+ self.idChecklists = blob['idChecklists']
+ if 'checklists' in blob:
+ self.checklists = blob['checklists']
+ else:
+ # TODO Only in Enterprise export, not available in board export
+ self.checklists = None
+
+ self.checklist_strs = []
+ self.change_history = []
+ self.column = None
+ self.final_comment_fields = {}
+
+
+ def figure_stuff_out(self, dao):
+ self.board_name = dao.get_board_name()
+ self.create_time_s = dao.figure_out_when_card_created(self)
+ self.column = dao.figure_out_first_card_column(self)
+ self.reporter = dao.guess_card_reporter(self, self.scrubber)
+ self.owner = dao.guess_card_owner(self, self.scrubber)
+ self.subscribers = dao.figure_out_all_subscribers(self, self.scrubber)
+ self.build_checklist_comment(dao)
+
+ for action in dao.get_relevant_actions(self.card_id):
+ self.handle_change(action, dao)
+
+ self.column = dao.get_column_name(self.idList)
+ # labels is text, idLabels is UIDs, append them to the end.
+ label_comment = ''
+ if self.labels:
+ label_comment = sorted(map(lambda k: k['color'], self.labels))
+ if self.idLabels:
+ for label_id in self.idLabels:
+ label_comment = ' ' + dao.get_label(label_id)
+ self.final_comment_fields['labels'] = label_comment
+
+ if self.due:
+ self.final_comment_fields['due'] = self.due
+
+ def build_checklist_comment(self, dao):
+ if not self.idChecklists:
+ return None
+ s = ''
+ if self.checklists is None:
+ log.warning('Failed to find checklists for card %s' % self.card_id)
+ return
+ for checklist in self.checklists:
+ s += 'Checklist: \n'
+ for item in checklist['checkItems']:
+ s+= ' * [%s] %s \n' % ('x' if item['state'] == 'complete' else
'', item['name'])
+ s += '\n'
+ change = {'type': 'comment', 'author': self.owner,
+ 'comment': s,
+ 'change_time_us':
s_to_us(parse_trello_ts_str(self.dateLastActivity))}
+ # SPage: cburroughs turns the checklist into a comment:
+ # self.change_history.append(change)
+ # SPage: instead, add to checklists string
+ self.checklist_strs.append(s)
+
+ def make_final_comment(self):
+ s = 'Trello Board: %s `%s` \n' % (self.board_name, self.idBoard)
+ s += "Trello Card: `%s` %s \n" % (self.card_id, self.url)
+ if len(self.final_comment_fields) > 0:
+ s += '\nExtra Info:\n'
+ for key in sorted(self.final_comment_fields):
+ s += ' * `%s`: `%s`\n' % (str(key),
unicode(self.final_comment_fields[key]))
+ return {'comment': s, 'ts_us': None}
+
+ def handle_change(self, j_change, dao):
+ if j_change['type'] == 'updateCard' and 'listBefore' in
j_change['data']:
+ change = {'type': 'custom-field',
+ 'author':
self.scrubber.get_phab_uid(dao.get_username(j_change['idMemberCreator'])),
+ 'key': 'std:maniphest:' + 'addthis:import-trello-column',
+ 'val':
dao.get_column_name(j_change['data']['listAfter']['id']),
+ 'change_time_us':
s_to_us(parse_trello_ts_str(j_change['date']))}
+ self.change_history.append(change)
+ # XXX This doesn't result in the card having the right column,
+ # elsewhere it's set to the column from the card's idList.
+ self.column =
dao.get_column_name(j_change['data']['listBefore']['id'])
+ elif j_change['type'] == 'commentCard':
+ change = {'type': 'comment',
+ 'author':
self.scrubber.get_phab_uid(dao.get_username(j_change['idMemberCreator'])),
+ 'comment': j_change['data']['text'],
+ 'change_time_us':
s_to_us(parse_trello_ts_str(j_change['date']))}
+ self.change_history.append(change)
+ elif j_change['type'] == 'updateCard' and 'closed' in
j_change['data']['card']:
+ phab_status = 'resolved' if j_change['data']['card']['closed']
else 'open'
+ change = {'type': 'status', 'author': self.owner,
+ 'status': phab_status,
+ 'change_time_us':
s_to_us(parse_trello_ts_str(j_change['date']))}
+ self.change_history.append(change)
+ self.closed = not j_change['data']['card']['closed']
+ elif j_change['type'] == 'updateCard' and 'old' in j_change['data']
and 'name' in j_change['data']['old']:
+ comment = 'Title change:\n * old: %s \n * new: %s' %
(j_change['data']['old']['name'], j_change['data']['card']['name'])
+ change = {'type': 'comment',
+ 'author':
self.scrubber.get_phab_uid(dao.get_username(j_change['idMemberCreator'])),
+ 'comment': comment,
+ 'change_time_us':
s_to_us(parse_trello_ts_str(j_change['date']))}
+ self.change_history.append(change)
+ elif j_change['type'] == 'updateCard' and 'old' in j_change['data']
and 'desc' in j_change['data']['old']:
+ comment = 'Desc change\n\n=== Old === \n\n %s \n\n=== New === \n\n
%s' % (j_change['data']['old']['desc'], j_change['data']['card']['desc'])
+ change = {'type': 'comment',
+ 'author':
self.scrubber.get_phab_uid(dao.get_username(j_change['idMemberCreator'])),
+ 'comment': comment,
+ 'change_time_us':
s_to_us(parse_trello_ts_str(j_change['date']))}
+ self.change_history.append(change)
+ elif j_change['type'] == 'updateCard' and 'old' in j_change['data']
and 'due' in j_change['data']['old']:
+ pass # Will just use the final due date
+ elif j_change['type'] == 'updateCard' and 'old' in j_change['data']
and 'pos' in j_change['data']['old']:
+ pass # just moving cards around in a list
+ else:
+ print j_change
+ log.warn('Unknown change condition type:%s id:%s for card %s' %
(j_change['type'], j_change['id'], self.card_id))
+
+ def to_transform_dict(self, import_project, task_id):
+ transform = {}
+ transform['must-preserve-id'] = False if task_id is None else True
+ transform['import-project'] = import_project
+ transform['base'] = {
+ 'ticket-id': self.card_id if task_id is None else task_id,
+ 'create-time-us': s_to_us(self.create_time_s),
+ 'owner': self.owner,
+ 'reporter': self.reporter,
+ 'summary': self.name,
+ 'description': self.desc,
+ 'priority': 50,
+ }
+ transform['init-custom'] = {}
+ transform['init-custom']['std:maniphest:' +
'addthis:import-trello-column'] = self.column
+ transform['changes'] = sorted(self.change_history, key=lambda d:
d['change_time_us'])
+ transform['final-comment'] = self.make_final_comment()
+ transform['final-subscribers'] = self.subscribers
+
+ return transform
+
+
+
+##### cmds #####
+
+def cmd_foo(args):
+ pass
+
+
+def cmd_print_labelnames(args):
+ board = TrelloDAO(args.trello_file)
+ pprint.pprint(board.get_labelnames())
+
+ pass
+
+def cmd_print_users(args):
+ board = TrelloDAO(args.trello_file)
+ pprint.pprint(board.get_usernames())
+
+ pass
+
+def cmd_print_user_map_test(args):
+ board = TrelloDAO(args.trello_file)
+ scrubber = TrelloScrubber('conf/trello-scrub.yaml')
+ for user in board.blob['members']:
+ print '%s <--> %s <--> %s' % (user['id'],
board.get_username(user['id']),
scrubber.get_phab_uid(board.get_username(user['id'])))
+
+def cmd_dump_cards(args):
+ mkdir_p('out/tickets')
+ board = TrelloDAO(args.trello_file)
+ scrubber = TrelloScrubber('conf/trello-scrub.yaml')
+ task_id = args.start_id
+ for j_card in board.blob['cards']:
+ card = TrelloCard(j_card, scrubber)
+ fname = os.path.join(args.dump_dir, card.card_id + '.json')
+ card.figure_stuff_out(board)
+ with open(fname, 'w') as f:
+ d = card.to_transform_dict(args.phab_project, task_id)
+ f.write(json.dumps(d, sort_keys=True,
+ indent=4, separators=(',', ': ')))
+ if task_id is not None:
+ task_id += 1
+
+##### main and friends #####
+
+def parse_args(argv):
+ def db_cmd(sub_p, cmd_name, cmd_help):
+ cmd_p = sub_p.add_parser(cmd_name, help=cmd_help)
+ cmd_p.add_argument('--log',
+ action='store', dest='log', default='stdout',
choices=['stdout', 'syslog', 'both'],
+ help='log to stdout and/or syslog')
+ cmd_p.add_argument('--log-level',
+ action='store', dest='log_level', default='WARNING',
+ choices=['CRITICAL', 'ERROR', 'WARNING', 'INFO',
'DEBUG', 'NOTSET'],
+ help='log to stdout and/or syslog')
+ cmd_p.add_argument('--log-facility',
+ action='store', dest='log_facility', default='user',
+ help='facility to use when using syslog')
+ cmd_p.add_argument('--trello-file',
+ action='store', dest='trello_file', required=True,
+ help='trello exported json file')
+
+ return cmd_p
+
+ parser = argparse.ArgumentParser(description="")
+ sub_p = parser.add_subparsers(dest='cmd')
+
+ foo_p = db_cmd(sub_p, 'foo', '')
+ foo_p.set_defaults(func=cmd_foo)
+
+ print_labelnames_p = db_cmd(sub_p, 'print-labelnames', '')
+ print_labelnames_p.set_defaults(func=cmd_print_labelnames)
+
+ print_users_p = db_cmd(sub_p, 'print-users', '')
+ print_users_p.set_defaults(func=cmd_print_users)
+
+ print_user_map_test_p = db_cmd(sub_p, 'print-user-map-test', '')
+ print_user_map_test_p.set_defaults(func=cmd_print_user_map_test)
+
+ dump_cards_p = db_cmd(sub_p, 'dump-cards', '')
+ dump_cards_p.set_defaults(func=cmd_dump_cards)
+ dump_cards_p.add_argument('--dump-dir',
+ action='store', dest='dump_dir',
default='out/tickets')
+ dump_cards_p.add_argument('--phab-project',
+ action='store', dest='phab_project',
required=True)
+ dump_cards_p.add_argument('--start-id', type=int,
+ action='store', dest='start_id')
+ dump_cards_p.set_defaults(func=cmd_dump_cards)
+
+ args = parser.parse_args(argv)
+ return args
+
+
+def setup_logging(handlers, facility, level):
+ global log
+
+ log = logging.getLogger('export-trac')
+ formatter = logging.Formatter(' | '.join(['%(asctime)s', '%(name)s',
'%(levelname)s', '%(message)s']))
+ if handlers in ['syslog', 'both']:
+ sh = logging.handlers.SysLogHandler(address='/dev/log',
facility=facility)
+ sh.setFormatter(formatter)
+ log.addHandler(sh)
+ if handlers in ['stdout', 'both']:
+ ch = logging.StreamHandler()
+ ch.setFormatter(formatter)
+ log.addHandler(ch)
+ lmap = {
+ 'CRITICAL': logging.CRITICAL,
+ 'ERROR': logging.ERROR,
+ 'WARNING': logging.WARNING,
+ 'INFO': logging.INFO,
+ 'DEBUG': logging.DEBUG,
+ 'NOTSET': logging.NOTSET
+ }
+ log.setLevel(lmap[level])
+
+
+def main(argv):
+ args = parse_args(argv)
+ try:
+ setup_logging(args.log, args.log_facility, args.log_level)
+ except Exception as e:
+ print >> sys.stderr, 'Failed to setup logging'
+ traceback.print_exc()
+ raise e
+
+ args.func(args)
+
+
+if __name__ == '__main__':
+ main(sys.argv[1:])
+
diff --git a/trello_create.py b/trello_create.py
index 5a4716e..2b7fe04 100644
--- a/trello_create.py
+++ b/trello_create.py
@@ -1,96 +1,298 @@
#
# Basic processor of Trello board export as JSON file.
# Documentation at
https://www.mediawiki.org/wiki/Phabricator/Trello_to_Phabricator
+# TODO 2015-01-25: log and write out the mapping card_id -> trello Tnnnn
+# TODO 2015-01-25: explicitly load data/phab-01 or phabricator.yaml file
instead of symlink
+# TODO 2015-01-25: figure out PROJ_phid, is it bust?
+# TODO 2015-01-25: imported https://phabricator.wikimedia.org/T87530 has " *
`labels`: ``", is this confusion or a bug with empty values?
+# TODO 2015-01-25: test with an actual label
+# TODO 2015-01-25: test with Ccs and Owners.
+
+
import argparse
import json
import sys
+import yaml
+
+# cburroughs' work
+from export_trello import TrelloCard, TrelloDAO, TrelloScrubber, setup_logging
from phabricator import Phabricator
+from wmfphablib import Phab as phabmacros # Cleaner (?), see
wmfphablib/phabapi.py
from wmfphablib import config
+from wmfphablib import log
+from wmfphablib import vlog
from wmfphablib import errorlog as elog
+class TrelloImporter:
+ # Map from Trello username to phabricator user.
+ userMapPhab01 = {
+ 'gtisza': 'Tgr',
+ "legoktm": "legoktm",
+ "matthewflaschen": "mattflaschen",
+ "pauginer": None,
+ "spage1": "spage",
+ }
+ userMapPhabWMF = {
+ # from mingleterminator.py
+ 'fflorin': 'Fabrice_Florin',
+ 'gdubuc': 'Gilles',
+ 'mholmquist': 'MarkTraceur',
+ 'gtisza': 'Tgr',
+ 'pginer': 'Pginer-WMF',
+ # From ack username
trabulous_dir/flow-current-iteration_lOh4XCy7_fm.json \
+ # | perl -pe 's/^\s+//' | sort | uniq
+ # Collaboration-Team members: Eloquence, DannyH,
Pginer-WMF, Spage, Quiddity, Mattflaschen, matthiasmullie, EBernhardson
+ "alexandermonk": None,
+ "antoinemusso": None,
+ "benmq": None,
+ "bsitu": None,
+ "dannyhorn1": "DannyH",
+ "erikbernhardson": "EBernhardson",
+ "jaredmzimmerman": None,
+ "jonrobson1": None,
+ "kaityhammerstein": None,
+ "legoktm": None,
+ "matthewflaschen": "mattflaschen",
+ "matthiasmullie": "matthiasmullie",
+ "maygalloway": None,
+ "moizsyed_": None,
+ "oliverkeyes": None,
+ "pauginer": "Pginer-WMF",
+ "quiddity1": "Quiddity",
+ "shahyarghobadpour": None,
+ "spage1": "Spage",
+ "wctaiwan": None,
+ "werdnum": None,
+ }
-def sanity_check(trello_json, data_file):
- if not 'cards' in trello_json:
- elog('No cards in input file %s' % (data_file))
- sys.exit(1)
+ def __init__(self, jsonName, args):
+ self.jsonName = jsonName
+ self.args = args # FIXME unpack all the args into member
variables?
+ self.verbose = args.verbose
+ if config.phab_host.find('phab-01') != -1:
+ self.host = 'phab-01'
+ elif config.phab_host.find('phabricator.wikimedia.org') != -1:
+ self.host = 'phabricator'
+ else:
+ self.json_error('Unrecognized host %s in config' %
(config.phab_host))
+ sys.exit(3)
+ self.board = TrelloDAO(self.jsonName)
+ trelloBoardName = self.board.get_board_name();
+ vlog('Trello board = %s' % (trelloBoardName))
-def get_board(trello_json, data_file):
- if 'name' in trello_json:
- return trello_json["name"]
- else:
- elog('No "name" for Trello board in input file %s' %
(data_file))
- sys.exit(2)
-
-# Determine projectPHID for the project name.
-def get_projectPHID(trelloProjectName):
- elog('FAKING IT WITH Flow (test) on phab-01.wmflabs.org!')
- # not implemented yet, need to run project.query see
- #
http://stackoverflow.com/questions/25753749/how-do-you-find-the-phid-of-a-phabricator-object
- return 'PHID-PROJ-ofxg2d73cpefxqnworfs'
-
-def create(card, dump_conduit):
- title = card["name"]
- # TODO: if Trello board is using scrum for Trello browser extension,
- # could extract story points /\s+\((\d+)\)' from card title to feed
into Sprint extension.
- desc = card["desc"]
- desc_tail = "\n--------------------------"
- desc_tail += "\n**Trello card**: [[ %s | %s ]]" % (card["url"],
card["shortLink"])
- full_description = desc + '\n' + desc_tail
- if dump_conduit:
- # This prints fields for maniphest.createtask
- print '"%s"\n"%s"\n\n' % (title.encode('unicode-escape'),
full_description.encode('unicode-escape'))
- else:
- elog('Have not actually implemented teh API calls yet :(')
+ self.projectPHID = self.get_projectPHID(self.args.phab_project);
+ vlog('Phabricator project PHID = %s' % (self.projectPHID))
-def process_cards(trello_json, dump_conduit):
- for card in trello_json["cards"]:
- # Skip archived cards ("closed" seems to correspond)
- if card["closed"]:
- continue
+ def connect_to_phab(self):
+ self.phab = Phabricator(config.phab_user,
+ config.phab_cert,
+ config.phab_host)
- # TODO: skip cards that are bugs
- # TODO: skip cards that already exist in Phab.
- create(card, dump_conduit)
+ self.phab.update_interfaces()
+ self.phabm = phabmacros('', '', '')
+ self.phabm.con = self.phab
+ # DEBUG to verify the API connection worked: print
phab.user.whoami()
+ vlog('Looks like we connected to the phab API \o/')
+
+
+ def sanity_check(self):
+ if not 'cards' in self.trelloJson:
+ self.json_error('No cards in input file')
+ sys.exit(1)
+
+ def testify(self, str):
+ if self.args.dry_run:
+ str = "TEST TEST run " + str
+
+ return str
+
+ def json_error(self, str):
+ elog('ERROR: %s in input file %s' % (str, self.jsonName))
+
+ # Determine projectPHID for the project name.
+ def get_projectPHID(self, trelloProjectName):
+ elog('FAKING IT WITH Flow (test) on phab-01.wmflabs.org!')
+ # not implemented yet, need to run project.query see
+ #
http://stackoverflow.com/questions/25753749/how-do-you-find-the-phid-of-a-phabricator-object
+ return 'PHID-PROJ-ofxg2d73cpefxqnworfs'
+
+ def get_trelloUserPHIDMap(self):
+ fname = 'data/trello_PHIDs_' + self.host + '.yaml'
+ with open(fname) as f:
+ yam = yaml.load(f)
+ if yam == None or len(yam) == 0:
+ elog('Warning: did not load trello->PHID mappings from
%s, did you run trello_makePHIDs.py?' % (fname))
+ else:
+ vlog('Looks like we loaded %d trello->PHID mappings
from %s' % (len(yam), fname))
+ return yam
+
+
+ # not implemented yet, need to run project.query see
+ #
http://stackoverflow.com/questions/25753749/how-do-you-find-the-phid-of-a-phabricator-object
+ return 'PHID-PROJ-ofxg2d73cpefxqnworfs'
+
+ # This is the workhorse
+ def createTask(self, card):
+ # Default some keys we always pass to createtask.
+ taskinfo = {
+ 'ownerPHID' : None,
+ 'ccPHIDs' : [],
+ 'projectPHIDs' : [self.projectPHID],
+ }
+
+ taskinfo["title"] = self.testify(card.name)
+
+ # TODO: if Trello board is using scrum for Trello browser
extension,
+ # could extract story points /\s+\((\d+)\)' from card title to
feed into Sprint extension.
+
+ # TODO: process attachments
+ # TODO: taskinfo["assignee"]
+ desc = self.testify(card.desc)
+
+ if card.checklist_strs:
+ desc += '\n' + '\n\n'.join(card.checklist_strs)
+ desc_tail = "\n--------------------------"
+ desc_tail += "\n**Trello card**: [[ %s | %s ]]" % (card.url,
card.shortLink)
+ desc_tail += "\nTrello column: " + card.column
+ if len(card.final_comment_fields) > 0:
+ s = ''
+ s += '\n'
+ for key in sorted(card.final_comment_fields):
+ s += ' * `%s`: `%s`\n' % (str(key),
unicode(card.final_comment_fields[key]))
+ desc_tail += s
+ #
+ # TODO: could add labels, label color, main attachment, etc.,
etc. to desc_tail.
+ taskinfo["description"] = desc + '\n' + desc_tail
+ # TODO: chasemp: what priority?
+ taskinfo['priority'] = 50
+ # TODO: chasemp: can I put "Trello lOh4XCy7" in "Reference"
field?
+
+ # Take the set of members
+ idMembers = card.idMembers
+ # Get the Trello username for the idMember
+ # memberNames = [ TrelloDAO.get_username(id) for id in
idMembers if TrelloDAO.get_username(id)]
+
+ if not 'FAILED' in card.owner:
+ taskinfo['ownerPHID'] = card.owner
+ taskinfo['ccPHIDs'] = [u for u in card.subscribers if not
'FAILED' in u]
+
+ # TODO: Add any other members with a PHID to the ccPHIDs
+ # TODO: Note remaining Trello members in desc_tail
+
+ vlog("card %s has members %s" % (card.shortLink, idMembers) )
+
+ # TODO: bugzilla_create.py and wmfphablib/phabapi.py use
axuiliary for
+ # BZ ref, but it doesn't work for Trello ref?
+ taskinfo["auxiliary"] =
{"std:maniphest:external_reference":"Trello %s" % (card.shortLink)}
+
+ if self.args.conduit:
+ # This prints fields for maniphest.createtask
+ print '"%s"\n"%s"\n\n' %
(taskinfo["title"].encode('unicode-escape'),
+
taskinfo["description"].encode('unicode-escape'))
+ else:
+ if self.args.dry_run:
+ log("dry-run to create a task for Trello card
%s ('%s')" %
+ (card.shortLink, taskinfo["title"]))
+ else:
+ ticket = self.phab.maniphest.createtask(
+
title = taskinfo['title'],
+
description = taskinfo['description'],
+
projectPHIDs = taskinfo['projectPHIDs'],
+
ownerPHID = taskinfo['ownerPHID'],
+
ccPHIDs = taskinfo['ccPHIDs'],
+
auxiliary = taskinfo['auxiliary']
+ )
+
+ log("Created task: T%s (%s)" % (ticket['id'],
ticket['phid']))
+
+
+ # Here bugzilla_create goes on to log actual creating
user and view/edit policy,
+ # then set_task_ctime to creation_time.
+
+ # Should I add comments to the card here,
+ # or a separate step that goes through action in
self.board.blob["actions"]
+ # handling type="commentCard"?
+
+
+ # Here are the types of objects in the "actions" array.
+ # 20 "type": "addAttachmentToCard",
+ # 9 "type": "addChecklistToCard",
+ # 2 "type": "addMemberToBoard",
+ # 38 "type": "addMemberToCard",
+ # 69 "type": "commentCard",
+ # 1 "type": "copyCard",
+ # 25 "type": "createCard",
+ # 3 "type": "createList",
+ # 6 "type": "deleteAttachmentFromCard",
+ # 29 "type": "moveCardFromBoard",
+ # 18 "type": "moveCardToBoard",
+ # 4 "type": "moveListFromBoard",
+ # 2 "type": "moveListToBoard",
+ # 3 "type": "removeChecklistFromCard",
+ # 14 "type": "removeMemberFromCard",
+ # 3 "type": "updateBoard",
+ # 698 "type": "updateCard",
+ # 48 "type": "updateCheckItemStateOnCard",
+ # 8 "type": "updateList",
+ # def getCardCreationMeta(self, cardId):
+ # Look around in JSON for ["actions"] array for member with
type:"createCard"
+ # with ["card"]["id"] = cardId
+ # and use the siblings ["date"] and ["memberCreator"]["id"]
+
+ # def getCardComments(self, cardId):
+ # Look around in JSON ["actions"] for member with
type:""commentCard"
+ # with ["card"]["id"] = cardId
+ # and use the siblings ["date"] and ["memberCreator"]["id"]
+
+ def process_cards(self):
+ self.connect_to_phab()
+
+ # Load PHID Map
+ self.trelloUserPHIDMap = self.get_trelloUserPHIDMap()
+
+ scrubber = TrelloScrubber('conf/trello-scrub.yaml')
+ for j_card in self.board.blob["cards"]:
+ card = TrelloCard(j_card, scrubber)
+ card.figure_stuff_out(self.board)
+ if self.args.column and not card.column ==
self.args.column:
+ continue
+ # Skip archived cards ("closed" seems to correspond?)
+ # But I think archive all cards in column doesn't set
this.
+ if card.closed:
+ continue
+
+ # TODO: skip cards that are bugs
+ # TODO: skip cards that already exist in Phab.
+ self.createTask(card)
def main():
parser = argparse.ArgumentParser()
parser.add_argument("-v", "--verbose", action="store_true",
- help="increase output verbosity")
- parser.add_argument("-j", "--json",
- help="Trello board JSON export file" )
- parser.add_argument("-d", "--dump_conduit", action="store_true",
- help="dump lines for maniphest.createtask")
- parser.add_argument("-p", "--project",
- help="Phabricator project for imported tasks (NOT
IMPLEMENTED YET")
+ help="increase output verbosity")
+ parser.add_argument("-vv", "--verbose-logging", action="store_true",
+ help="wmfphablib verbose logging")
+ parser.add_argument("-j", "--json", required=True,
+ help="Trello board JSON export file" )
+ parser.add_argument("-c", "--conduit", action="store_true",
+ help="print out lines suitable for conduit
maniphest.createtask")
+ parser.add_argument("-d", "--dry-run", action="store_true",
+ help="don't actually add anything to Phabricator")
+ parser.add_argument("-l", "--column",
+ help="Name the one column to import")
+ parser.add_argument("-t", "--test-run", action="store_true",
+ help="prefix titles and description with 'TEST
trabuloust TEST' disclaimers")
+ parser.add_argument("-p", "--phab_project",
+ help="Phabricator project for imported tasks (NOT
IMPLEMENTED YET")
args = parser.parse_args()
- with open(args.json) as data_file:
- trello_json = json.load(data_file)
- sanity_check(trello_json, data_file);
-
- # From python-phabricator/README.rst
- phab = Phabricator(config.phab_user,
- config.phab_cert,
- config.phab_host)
-
- phab.update_interfaces()
- # DEBUG to verify the API connection worked: print phab.user.whoami()
- if args.verbose:
- elog('Looks like we connected to the phab API \o/')
-
- trelloBoard = get_board(trello_json, args.project);
- if args.verbose:
- elog('Trello board = %s' % (trelloBoard))
-
- projectPHID = get_projectPHID(args.project);
- if args.verbose:
- elog('Phabricator project PHID = %s' % (projectPHID))
-
- process_cards(trello_json, args.dump_conduit);
+ # As used in export_trello.py functions.
+ setup_logging('stdout', 'user', 'WARNING')
+ trell = TrelloImporter(args.json, args)
+ trell.process_cards()
if __name__ == '__main__':
- main()
+ main()
--
To view, visit https://gerrit.wikimedia.org/r/188287
To unsubscribe, visit https://gerrit.wikimedia.org/r/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I70897def2a781191fe06213cf157284f4e29b1a6
Gerrit-PatchSet: 1
Gerrit-Project: phabricator/tools
Gerrit-Branch: master
Gerrit-Owner: Spage <[email protected]>
_______________________________________________
MediaWiki-commits mailing list
[email protected]
https://lists.wikimedia.org/mailman/listinfo/mediawiki-commits
