Hi all, Seems like this github_jira_sync.py <https://github.com/apache/spark/blob/master/dev/github_jira_sync.py> script seems stopped working completely now.
https://issues.apache.org/jira/browse/SPARK-31532 <> https://github.com/apache/spark/pull/28316 https://issues.apache.org/jira/browse/SPARK-31529 <> https://github.com/apache/spark/pull/28315 https://issues.apache.org/jira/browse/SPARK-31528 <> https://github.com/apache/spark/pull/28313 Josh, would you mind taking a look please when you find some time? There is a bunch of JIRAs now, and it is very confusing which JIRA is in progress with a PR or not. 2019년 7월 26일 (금) 오후 1:20, Hyukjin Kwon <gurwls...@gmail.com>님이 작성: > Just FYI, I had to come up with a better JQL to filter out the JIRAs that > already have linked PRs. > In case it helps someone, I use this JQL now to look through the open > JIRAs: > > project = SPARK AND > status = Open AND > NOT issueFunction in linkedIssuesOfRemote("Github Pull Request *") > ORDER BY created DESC, priority DESC, updated DESC > > > > > 2019년 7월 19일 (금) 오후 4:54, Hyukjin Kwon <gurwls...@gmail.com>님이 작성: > >> That's a great explanation. Thanks I didn't know that. >> >> Josh, do you know who I should ping on this? >> >> On Fri, 19 Jul 2019, 16:52 Dongjoon Hyun, <dongjoon.h...@gmail.com> >> wrote: >> >>> Hi, Hyukjin. >>> >>> In short, there are two bots. And, the current situation happens when >>> only one bot with `dev/github_jira_sync.py` works. >>> >>> And, `dev/github_jira_sync.py` is irrelevant to the JIRA status change >>> because it only use `add_remote_link` and `add_comment` API. >>> I know only this bot (in Apache Spark repository repo) >>> >>> AFAIK, `deb/github_jira_sync.py`'s activity is done under JIRA ID >>> `githubbot` (Name: `ASF GitHub Bot`). >>> And, the other bot's activity is done under JIRA ID `apachespark` (Name: >>> `Apache Spark`). >>> The other bot is the one which Josh mentioned before. (in >>> `databricks/spark-pr-dashboard` repo). >>> >>> The root cause will be the same. The API key used by the bot is rejected >>> by Apache JIRA and forwarded to CAPCHAR. >>> >>> Bests, >>> Dongjoon. >>> >>> On Thu, Jul 18, 2019 at 8:24 PM Hyukjin Kwon <gurwls...@gmail.com> >>> wrote: >>> >>>> Hi all, >>>> >>>> Seems this issue is re-happening again. Seems the PR link is properly >>>> created in the corresponding JIRA but it doesn't change the JIRA's status >>>> from OPEN to IN-PROGRESS. >>>> >>>> See, for instance, >>>> >>>> https://issues.apache.org/jira/browse/SPARK-28443 >>>> https://issues.apache.org/jira/browse/SPARK-28440 >>>> https://issues.apache.org/jira/browse/SPARK-28436 >>>> https://issues.apache.org/jira/browse/SPARK-28434 >>>> https://issues.apache.org/jira/browse/SPARK-28433 >>>> https://issues.apache.org/jira/browse/SPARK-28431 >>>> >>>> Josh and Dongjoon, do you guys maybe have any idea? >>>> >>>> 2019년 4월 25일 (목) 오후 3:09, Hyukjin Kwon <gurwls...@gmail.com>님이 작성: >>>> >>>>> Thank you so much Josh .. !! >>>>> >>>>> 2019년 4월 25일 (목) 오후 3:04, Josh Rosen <rosenvi...@gmail.com>님이 작성: >>>>> >>>>>> The code for this runs in http://spark-prs.appspot.com (see >>>>>> https://github.com/databricks/spark-pr-dashboard/blob/1e799c9e510fa8cdc9a6c084a777436bebeabe10/sparkprs/controllers/tasks.py#L137 >>>>>> ) >>>>>> >>>>>> I checked the AppEngine logs and it looks like we're getting error >>>>>> responses, possibly due to a credentials issue: >>>>>> >>>>>> Exception when starting progress on JIRA issue SPARK-27355 ( >>>>>>> /base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py:142 >>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=142&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>) >>>>>>> Traceback (most recent call last): File >>>>>>> Traceback (most recent call last): >>>>>>> File >>>>>>> "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/controllers/tasks.py", >>>>>>> line 138 >>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fcontrollers%2Ftasks.py&line=138&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>, >>>>>>> in update_pr start_issue_progress("%s-%s" % (app.config['JIRA_PROJECT'], >>>>>>> issue_number)) File >>>>>>> start_issue_progress("%s-%s" % (app.config['JIRA_PROJECT'], >>>>>>> issue_number)) >>>>>>> File >>>>>>> "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py", >>>>>>> line 27 >>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=27&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>, >>>>>>> in start_issue_progress jira_client = get_jira_client() File >>>>>>> jira_client = get_jira_client() >>>>>>> File >>>>>>> "/base/data/home/apps/s~spark-prs/live.412416057856832734/sparkprs/jira_api.py", >>>>>>> line 18 >>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Fsparkprs%2Fjira_api.py&line=18&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>, >>>>>>> in get_jira_client app.config['JIRA_PASSWORD'])) File >>>>>>> app.config['JIRA_PASSWORD'])) >>>>>>> File >>>>>>> "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py", >>>>>>> line 472 >>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=472&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>, >>>>>>> in __init__ si = self.server_info() File >>>>>>> si = self.server_info() >>>>>>> File >>>>>>> "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py", >>>>>>> line 2133 >>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2133&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>, >>>>>>> in server_info j = self._get_json('serverInfo') File >>>>>>> j = self._get_json('serverInfo') >>>>>>> File >>>>>>> "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/client.py", >>>>>>> line 2549 >>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fclient.py&line=2549&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>, >>>>>>> in _get_json r = self._session.get(url, params=params) File >>>>>>> r = self._session.get(url, params=params) >>>>>>> File >>>>>>> "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py", >>>>>>> line 151 >>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=151&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>, >>>>>>> in get return self.__verb('GET', url, **kwargs) File >>>>>>> return self.__verb('GET', url, **kwargs) >>>>>>> File >>>>>>> "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py", >>>>>>> line 147 >>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=147&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>, >>>>>>> in __verb raise_on_error(response, verb=verb, **kwargs) File >>>>>>> raise_on_error(response, verb=verb, **kwargs) >>>>>>> File >>>>>>> "/base/data/home/apps/s~spark-prs/live.412416057856832734/lib/jira/resilientsession.py", >>>>>>> line 57 >>>>>>> <https://console.cloud.google.com/debug/fromlog?appModule=default&appVersion=live&file=%2Fbase%2Fdata%2Fhome%2Fapps%2Fs~spark-prs%2Flive.412416057856832734%2Flib%2Fjira%2Fresilientsession.py&line=57&logInsertId=5cc1483600029309a7af76d5&logNanos=1556170805012269000&nestedLogIndex=3&project=spark-prs&src=ac>, >>>>>>> in raise_on_error r.status_code, error, r.url, request=request, >>>>>>> response=r, >>>>>>> **kwargs) JIRAError: JiraError HTTP 403 url: >>>>>>> https://issues.apache.org/jira/rest/api/2/serverInfo text: >>>>>>> CAPTCHA_CHALLENGE; login-url= >>>>>>> https://issues.apache.org/jira/login.jsp r.status_code, error, >>>>>>> r.url, request=request, response=r, **kwargs) >>>>>>> JIRAError: JiraError HTTP 403 url: >>>>>>> https://issues.apache.org/jira/rest/api/2/serverInfo >>>>>>> text: CAPTCHA_CHALLENGE; login-url= >>>>>>> https://issues.apache.org/jira/login.jsp >>>>>> >>>>>> >>>>>> It looks like ASF JIRA was throwing a captcha challenge at us, so I >>>>>> used the credentials to manually log in and complete the challenge. >>>>>> >>>>>> Hopefully that's enough to fix things, but to prevent re-occurrence >>>>>> we might need to change the login credential type from username + >>>>>> password >>>>>> to instead use OAuth tokens. >>>>>> >>>>>> On Wed, Apr 24, 2019 at 10:42 PM Hyukjin Kwon <gurwls...@gmail.com> >>>>>> wrote: >>>>>> >>>>>>> Can anyone take a look for this one? OPEN status JIRAs are being >>>>>>> rapidly increased (from around 2400 to 2600) >>>>>>> >>>>>>> 2019년 4월 19일 (금) 오후 8:05, Hyukjin Kwon <gurwls...@gmail.com>님이 작성: >>>>>>> >>>>>>>> Hi all, >>>>>>>> >>>>>>>> Looks 'spark/dev/github_jira_sync.py' is not running correctly >>>>>>>> somewhere. >>>>>>>> Usually the JIRA's status should be updated to "IN PROGRESS" when >>>>>>>> somebody opens a PR against a JIRA. >>>>>>>> Looks now it only leaves a link and does not change JIRA's status. >>>>>>>> >>>>>>>> Can someone else who knows where it's running can check this? >>>>>>>> >>>>>>>> FWIW, I check every PR and JIRA almost every day but ever since >>>>>>>> this happened, this makes (at least to me) duplicately check the JIRAs. >>>>>>>> Previously, if I check all the PRs and JIRAs, they were not >>>>>>>> duplicated because JIRAs having PRs have different status, "IN >>>>>>>> PROGRESS" >>>>>>>> but now all JIRAs have "OPEN" status. >>>>>>>> >>>>>>>> Thanks. >>>>>>>> >>>>>>>
