I started to play around with the (unofficial) Issue Import API <https://gist.github.com/jonmagic/5282384165e0f86ef105> and so far so good. - Original timestamps (created, updated) are preserved - An issue and its comments can be imported with one post request; importing is done asynchronously but the background job looks sufficiently fast. - Notifications are not triggered with mentions in the issue comments (e.g. https://github.com/mocobeta/sandbox-lucene-10557/issues/249)
I will do more tests and rewrite the scripts with this API and if it works as expected, try to migrate whole Jira issues into a test repo next time; it still needs two-pass migration but could be done in a few hours with the default rate limit. Is there anybody who kindly provides a github account to make sure that "any notifications are never triggered" when executing migration? I confirmed this with my account, just wanted to test it with other one or two accounts. Tomoko 2022年6月24日(金) 21:12 Adrien Grand <jpou...@gmail.com>: > FWIW I don't mind receiving these one-time notifications for the purpose > of the migration. > > On Fri, Jun 24, 2022 at 12:58 PM Tomoko Uchida < > tomoko.uchida.1...@gmail.com> wrote: > >> > If you could give me some "example mail", I can add a "feed to trash" >> sieve rule before you start it. >> >> If we decide to migrate all Jira issues to GItHub with reporter/author >> account re-mapping (hyperlinks), and we still cannot find a workaround >> not to trigger notifications, I'll let you all know the example mail >> text. >> I can also trigger some real notifications by simulating a few >> migrations for people who wish to know what will happen in their >> mailbox/github UI beforehand. I'm really not sure what will happen to >> everyone by triggering so many notifications. >> >> Precisely speaking, you will receive the same number of notifications >> to your all activities in Jira so far: reporting an issue, assigning >> yourself to an issue, and leaving a comment. >> GitHub UI and mailers will aggregate them into "issues", I think. >> >> > Another question: With the alternative API is it possible to keep >> original Reporters and Authors of comments, too? >> >> I haven't tried the unofficial API yet, but as far as I read the >> examples, it is not possible to keep original reporters or authors. >> You cannot change/tweak GitHub issue reporter or author by any means - >> it looks a sane decision to me. >> >> >> 2022年6月24日(金) 16:18 Uwe Schindler <u...@thetaphi.de>: >> > >> > Hi, >> > >> > Am 23.06.2022 um 19:09 schrieb Tomoko Uchida: >> > >> > uschindler,Uwe Schindler,2474 >> > >> > If you could give me some "example mail", I can add a "feed to trash" >> sieve rule before you start it. >> > >> > We have 109158 resources (10608 issues + 98550 comments) in total. With >> the default rate limit of 5000 reqs per hour, it will take 22 hours if >> there are no failures/retries. With an enterprise account, it will take 7-8 >> hours I think. >> > >> > So a sieve rule would be good to not get constant mail all the time, >> because my 2474 issues (or notifications?) will be distrubuted over the >> whole day. >> > >> > Another question: With the alternative API is it possible to keep >> original Reporters and Authors of comments, too? >> > >> > Uwe >> > >> > Tomoko >> > >> > >> > 2022年6月24日(金) 0:39 Tomoko Uchida <tomoko.uchida.1...@gmail.com>: >> >> >> >> It seems not new - I don't figure out why this is not published as a >> public API yet but according to the comments there, it could be >> buggy/unstable (still worth a try to me). >> >> >> >> >> >> >> >> 2022年6月24日(金) 0:26 Tomoko Uchida <tomoko.uchida.1...@gmail.com>: >> >>> >> >>> I just browsed through this article about the "import issues" API >> (looks pretty new and on technical preview status?) >> >>> https://gist.github.com/jonmagic/5282384165e0f86ef105 >> >>> >> >>> Seems it solves many of our considerations - preserving original >> timestamp, bulk importing issue comments, and not triggering notifications. >> >>> >> >>> I'll try it later; thank you Rob for providing the information. >> >>> >> >>> >> >>> 2022年6月23日(木) 23:18 Michael Sokolov <msoko...@gmail.com>: >> >>>> >> >>>> oh phew! glad to hear this was expected >> >>>> >> >>>> On Thu, Jun 23, 2022 at 10:17 AM Tomoko Uchida >> >>>> <tomoko.uchida.1...@gmail.com> wrote: >> >>>> > >> >>>> > > Many comments were lost in the transfer. The last one in the >> copy is >> >>>> > > only about 1/4 of the way through this gigantic issue. This >> really is >> >>>> > > a blocker I think. >> >>>> > >> >>>> > I limited the number of comments per issue up to ten for testing. >> We can migrate literally all comments - one by one. >> >>>> > >> >>>> > Tomoko >> >>>> > >> >>>> > >> >>>> > 2022年6月23日(木) 23:09 Tomoko Uchida <tomoko.uchida.1...@gmail.com>: >> >>>> >> >> >>>> >> Hi, >> >>>> >> I have little now to carefully read/respond to all replies right >> now, but just wanted to answer this. >> >>>> >> >> >>>> >> > Will it be possible to preserve links from issues -> pull >> requests? >> >>>> >> >> >>>> >> Yes it's a bit cumbersome (and it could be difficult to make sure >> that all links to PRs are covered - it's not solid metadata, you need to >> parse github bot's comments) but surely possible. >> >>>> >> See https://github.com/mocobeta/sandbox-lucene-10557/issues/188. >> >>>> >> >> >>>> >> As for notifications and attachment files, if there are ways to >> manage these it'd be great. >> >>>> >> >> >>>> >> >> >>>> >> 2022年6月23日(木) 22:59 Dawid Weiss <dawid.we...@gmail.com>: >> >>>> >>> >> >>>> >>> >> >>>> >>> Interesting, thanks Rob. I see the attachments have been ported >> in that article as well - something the official API is not able to do. >> >>>> >>> >> >>>> >>> https://jira.spring.io/browse/DATACMNS-617?redirect=false >> >>>> >>> >> https://github.com/spring-projects/spring-data-commons/issues/1080 >> >>>> >>> >> >>>> >>> Dawid >> >>>> >>> >> >>>> >>> On Thu, Jun 23, 2022 at 3:26 PM Rob Audenaerde < >> rob.audenae...@gmail.com> wrote: >> >>>> >>>> >> >>>> >>>> I didn't read the entire thread, so apologies if this is a >> duplicate: >> >>>> >>>> >> >>>> >>>> Did you check >> https://spring.io/blog/2021/01/07/spring-data-s-migration-from-jira-to-github-issues >> >>>> >>>> >> >>>> >>>> They especially write there is an api that doesn't trigger >> notifications. >> >>>> >>>> >> >>>> >>>> It is documented here: >> https://gist.github.com/jonmagic/5282384165e0f86ef105 >> >>>> >>>> >> >>>> >>>> >> >>>> >>>> >> >>>> >>>> >> >>>> >>>> >> >>>> >>>> >> >>>> >>>> On Thu, Jun 23, 2022, 15:14 Michael Sokolov <msoko...@gmail.com> >> wrote: >> >>>> >>>>> >> >>>> >>>>> Yes thank you! You say this is not difficult, but it looks >> like a big >> >>>> >>>>> job to me! Here are a bunch of things I noticed that we would >> ideally >> >>>> >>>>> address (from looking at one long and complex issue, >> LUCENE-9004). I >> >>>> >>>>> wouldn't be so bold as to say these should block us from >> proceeding if >> >>>> >>>>> they're not addressed, just want to point out there is >> potentially a >> >>>> >>>>> lot to do: >> >>>> >>>>> >> >>>> >>>>> Will it be possible to preserve links from issues -> pull >> requests? >> >>>> >>>>> That seems like one of the most important pieces of "metadata". >> >>>> >>>>> >> >>>> >>>>> As far as attached files go, I see you seem to have made an >> attempt? >> >>>> >>>>> There is a link in >> https://issues.apache.org/jira/browse/LUCENE-9004 >> >>>> >>>>> where you had posted a picture of a graph, for example; in >> >>>> >>>>> https://github.com/mocobeta/sandbox-lucene-10557/issues/171 >> it is >> >>>> >>>>> represented as a link. When you click on the link you get an >> error >> >>>> >>>>> though. I wonder if it would be possible to link back to the >> images >> >>>> >>>>> hosted in JIRA? (Ideally as an <IMG> tag; otherwise a link >> would be >> >>>> >>>>> good). >> >>>> >>>>> >> >>>> >>>>> I agree with Ryan - I'd be willing to bulk-delete 1000 >> notifications >> >>>> >>>>> if it means preserving hyperlinks to people >> >>>> >>>>> >> >>>> >>>>> Numbered list formatting became giant bold text (see the >> comment >> >>>> >>>>> containing "Here's a strategy") >> >>>> >>>>> >> >>>> >>>>> Many comments were lost in the transfer. The last one in the >> copy is >> >>>> >>>>> only about 1/4 of the way through this gigantic issue. This >> really is >> >>>> >>>>> a blocker I think. I wonder what happened? Maybe some API >> calls failed >> >>>> >>>>> and we need to retry??? >> >>>> >>>>> >> >>>> >>>>> I wanted to check other fancy formatting (tables, block >> comments, code >> >>>> >>>>> blocks, etc) but haven't looked at these yet... >> >>>> >>>>> >> >>>> >>>>> On Wed, Jun 22, 2022 at 8:34 AM Ryan Ernst <r...@iernst.net> >> wrote: >> >>>> >>>>> > >> >>>> >>>>> > This is great work Tomoko! A couple minor thoughts: >> >>>> >>>>> > >> >>>> >>>>> > * I don’t think a flood of notifications from the import is >> a problem. It’s a one time hassle, and having the actual user links is nice >> for GitHub’s cross linking system. >> >>>> >>>>> > >> >>>> >>>>> > * Do you have an estimate for how many api calls are needed? >> How many total issues+comments exist in jira? I assume the limits you dealt >> with were the default 5k requests per hour. If that will take too long, we >> could consider using a user from an enterprise account which has 3x the >> limit. >> >>>> >>>>> > >> >>>> >>>>> > On Tue, Jun 21, 2022 at 15:56 Tomoko Uchida < >> tomoko.uchida.1...@gmail.com> wrote: >> >>>> >>>>> >> >> >>>> >>>>> >> Hi all, >> >>>> >>>>> >> again - this is about GitHub migration. >> >>>> >>>>> >> >> >>>> >>>>> >> We have a large disagreement on whether we should migrate >> existing Jira issues (including all closed issues) to GitHub or not. >> >>>> >>>>> >> >> >>>> >>>>> >> I drafted a tiny migration tool [1] to see how it looks if >> we move Jira issues to GitHub, and tried to migrate a small portion of Jira >> issues/comments to a test repo. You can see it here: >> >>>> >>>>> >> - Closed issues list >> https://github.com/mocobeta/sandbox-lucene-10557/issues?q=is%3Aissue+is%3Aclosed >> >>>> >>>>> >> - Unresolved issues list: >> https://github.com/mocobeta/sandbox-lucene-10557/issues >> >>>> >>>>> >> >> >>>> >>>>> >> I don't deserve to have a strong opinion on how we should >> treat 20+ years of history so I would reserve my opinion - would the >> prototype be of some help to have a conversation? >> >>>> >>>>> >> I have to leave for a while, I'd be glad if you have a talk >> on it while I'm away and hopefully reach an agreement. >> >>>> >>>>> >> >> >>>> >>>>> >> Here's a summary of what can be done. >> >>>> >>>>> >> >> >>>> >>>>> >> You can: >> >>>> >>>>> >> * migrate all texts in issue descriptions and comments to >> GitHub; browsing/searching old issues should work fine. >> >>>> >>>>> >> * extract every issue metadata from Jira and port it to >> labels or issue descriptions (as plain text). >> >>>> >>>>> >> * map Jira cross-issue link "LUCENE-xxx" to GitHub issue >> mention "#yyy". >> >>>> >>>>> >> * see this example: >> https://github.com/mocobeta/sandbox-lucene-10557/issues/218 >> >>>> >>>>> >> * map Jira user ids to GitHub accounts if the mapping is >> given. >> >>>> >>>>> >> * convert Jira markups to Markdown with parser library. >> >>>> >>>>> >> * not perfect - there can be many conversion errors >> >>>> >>>>> >> >> >>>> >>>>> >> And here are the limitations. (Please correct me if I'm >> missing something.) >> >>>> >>>>> >> >> >>>> >>>>> >> You cannot: >> >>>> >>>>> >> * simulate original authors and timestamps; they have to be >> preserved in free-text forms. >> >>>> >>>>> >> * migrate attached files (patches, images, etc.) to GitHub; >> these have to remain in Jira. >> >>>> >>>>> >> * it's not allowed to programmatically upload files and >> attach them to issues. >> >>>> >>>>> >> * create hyperlinks from issues to GitHub accounts >> (reporters, comment authors, etc.) by mentions; otherwise everyone will >> receive a huge volume of notifications. >> >>>> >>>>> >> * still accounts can be noted with a markup `@xxxx` >> (without mentioning) in their right place >> >>>> >>>>> >> * "bulk" import issues/comments. Each resource has to be >> posted one by one. Migration would take many hours (perhaps days?) due to >> the severe API call rate limit. >> >>>> >>>>> >> >> >>>> >>>>> >> It's not a particularly difficult task, however, there will >> be other uncontrollable things I haven't noticed yet. >> >>>> >>>>> >> >> >>>> >>>>> >> [1] >> https://github.com/mocobeta/sandbox-lucene-10557/tree/main/migration >> >>>> >>>>> >> >> >>>> >>>>> >> Tomoko >> >>>> >>>>> >> >>>> >>>>> >> --------------------------------------------------------------------- >> >>>> >>>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> >>>> >>>>> For additional commands, e-mail: dev-h...@lucene.apache.org >> >>>> >>>>> >> >>>> >> >>>> --------------------------------------------------------------------- >> >>>> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> >>>> For additional commands, e-mail: dev-h...@lucene.apache.org >> >>>> >> > -- >> > Uwe Schindler >> > Achterdiek 19, D-28357 Bremen >> > https://www.thetaphi.de >> > eMail: u...@thetaphi.de >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org >> For additional commands, e-mail: dev-h...@lucene.apache.org >> >> > > -- > Adrien >