[
https://issues.apache.org/jira/browse/BEAM-5240?focusedWorklogId=155595&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-155595
]
ASF GitHub Bot logged work on BEAM-5240:
----------------------------------------
Author: ASF GitHub Bot
Created on: 17/Oct/18 20:36
Start Date: 17/Oct/18 20:36
Worklog Time Spent: 10m
Work Description: swegner closed pull request #6711: [BEAM-5240] Add Jira
data to Beam post-commits dashboard
URL: https://github.com/apache/beam/pull/6711
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/.test-infra/metrics/README.md b/.test-infra/metrics/README.md
index dca8d17d02c..d18c52c4c35 100644
--- a/.test-infra/metrics/README.md
+++ b/.test-infra/metrics/README.md
@@ -102,13 +102,22 @@ docker push gcr.io/${PROJECT_ID}/beammetricssyncjenkins:v1
## Kubernetes update
https://kubernetes.io/docs/concepts/workloads/controllers/deployment/
-1. Build and publish sync containers
```sh
+# Build and publish sync containers
cd sync/jenkins
docker build -t gcr.io/${PROJECT_ID}/beammetricssyncjenkins:v1 .
docker push -t gcr.io/${PROJECT_ID}/beammetricssyncjenkins:v1
+
+# If needed check current pod status
+kubectl get pods
+kubectl describe pod <pod_id>
+
+# Update container image via one of the following.
+## update image for container
+kubectl set image deployment/beamgrafana container=<new_image_name>
+## or update deployemnt from yaml file
+kubectl replace -f beamgrafana-deploy.yaml
```
-1. Update image for container `kubectl set image deployment/beamgrafana
container=<new_image_name>`
## Useful Kubernetes commands and hints
diff --git a/.test-infra/metrics/beamgrafana-deploy.yaml
b/.test-infra/metrics/beamgrafana-deploy.yaml
index 62379510371..34941284172 100644
--- a/.test-infra/metrics/beamgrafana-deploy.yaml
+++ b/.test-infra/metrics/beamgrafana-deploy.yaml
@@ -33,36 +33,6 @@ spec:
securityContext:
fsGroup: 1000
containers:
- - name: beammetricssyncjenkins
- image: gcr.io/apache-beam-testing/beammetricssyncjenkins:v15
- env:
- - name: JENSYNC_HOST
- value: 127.0.0.1
- - name: JENSYNC_PORT
- value: "5432"
- - name: JENSYNC_DBNAME
- value: beammetrics
- - name: JENSYNC_DBUSERNAME
- valueFrom:
- secretKeyRef:
- name: beammetrics-psql-db-credentials
- key: username
- - name: JENSYNC_DBPWD
- valueFrom:
- secretKeyRef:
- name: beammetrics-psql-db-credentials
- key: password
- - name: cloudsql-proxy
- image: gcr.io/cloudsql-docker/gce-proxy:1.11
- command: ["/cloud_sql_proxy",
-
"-instances=apache-beam-testing:us-west2:beammetrics=tcp:5432"]
- env:
- - name: GOOGLE_APPLICATION_CREDENTIALS
- value: /secrets/cloudsql/config.json
- volumeMounts:
- - name: beammetrics-psql-credentials
- mountPath: /secrets/cloudsql
- readOnly: true
- name: beamgrafana
image: grafana/grafana
securityContext:
@@ -99,6 +69,55 @@ spec:
name: beam-grafana-etcdata
- mountPath: /var/log/grafana
name: beam-grafana-logdata
+ - name: cloudsql-proxy
+ image: gcr.io/cloudsql-docker/gce-proxy:1.11
+ command: ["/cloud_sql_proxy",
+
"-instances=apache-beam-testing:us-west2:beammetrics=tcp:5432"]
+ env:
+ - name: GOOGLE_APPLICATION_CREDENTIALS
+ value: /secrets/cloudsql/config.json
+ volumeMounts:
+ - name: beammetrics-psql-credentials
+ mountPath: /secrets/cloudsql
+ readOnly: true
+ - name: beammetricssyncjenkins
+ image: gcr.io/apache-beam-testing/beammetricssyncjenkins:v20181016
+ env:
+ - name: JENSYNC_HOST
+ value: 127.0.0.1
+ - name: JENSYNC_PORT
+ value: "5432"
+ - name: JENSYNC_DBNAME
+ value: beammetrics
+ - name: JENSYNC_DBUSERNAME
+ valueFrom:
+ secretKeyRef:
+ name: beammetrics-psql-db-credentials
+ key: username
+ - name: JENSYNC_DBPWD
+ valueFrom:
+ secretKeyRef:
+ name: beammetrics-psql-db-credentials
+ key: password
+ - name: beammetricssyncjira
+ image: gcr.io/apache-beam-testing/beammetricssyncjira:v20181016
+ env:
+ - name: DB_HOST
+ value: 127.0.0.1
+ - name: DB_PORT
+ value: "5432"
+ - name: DB_DBNAME
+ value: beammetrics
+ - name: DB_DBUSERNAME
+ valueFrom:
+ secretKeyRef:
+ name: beammetrics-psql-db-credentials
+ key: username
+ - name: DB_DBPWD
+ valueFrom:
+ secretKeyRef:
+ name: beammetrics-psql-db-credentials
+ key: password
volumes:
- name: beammetrics-psql-credentials
secret:
diff --git a/.test-infra/metrics/docker-compose.yml
b/.test-infra/metrics/docker-compose.yml
index 39a018ff519..95d70c0cad8 100644
--- a/.test-infra/metrics/docker-compose.yml
+++ b/.test-infra/metrics/docker-compose.yml
@@ -55,6 +55,18 @@ services:
- JENSYNC_DBNAME=beam_metrics
- JENSYNC_DBUSERNAME=admin
- JENSYNC_DBPWD=<PGPasswordHere>
+ syncjira:
+ image: syncjira
+ container_name: beamsyncjira
+ build:
+ context: ./sync/jira
+ dockerfile: Dockerfile
+ environment:
+ - DB_HOST=beampostgresql
+ - DB_PORT=5432
+ - DB_DBNAME=beam_metrics
+ - DB_DBUSERNAME=admin
+ - DB_DBPWD=<PGPasswordHere>
volumes:
beam-postgresql-data:
beam-grafana-libdata:
diff --git a/.test-infra/metrics/sync/jenkins/syncjenkins.py
b/.test-infra/metrics/sync/jenkins/syncjenkins.py
index 7d5d1d2360c..0f198cc132d 100644
--- a/.test-infra/metrics/sync/jenkins/syncjenkins.py
+++ b/.test-infra/metrics/sync/jenkins/syncjenkins.py
@@ -97,7 +97,7 @@ def initDbTablesIfNeeded():
connection.close()
-def fetchSyncedJobsBuildVersions(cursor):
+def fetchLastSyncTimestamp(cursor):
fetchQuery = f'''
select job_name, max(build_id)
from {jenkinsBuildsTableName}
@@ -152,7 +152,7 @@ def insertRow(cursor, rowValues):
def fetchNewData():
connection = initConnection()
cursor = connection.cursor()
- syncedJobs = fetchSyncedJobsBuildVersions(cursor)
+ syncedJobs = fetchLastSyncTimestamp(cursor)
cursor.close()
connection.close()
@@ -197,7 +197,7 @@ def probeJenkinsIsUp():
while True:
if not probeJenkinsIsUp():
- print("Jenkins is unavailabel, skipping fetching data.")
+ print("Jenkins is unavailable, skipping fetching data.")
continue
else:
fetchNewData()
diff --git a/.test-infra/metrics/sync/jira/Dockerfile
b/.test-infra/metrics/sync/jira/Dockerfile
new file mode 100644
index 00000000000..afd681e212e
--- /dev/null
+++ b/.test-infra/metrics/sync/jira/Dockerfile
@@ -0,0 +1,28 @@
+################################################################################
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements. See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership. The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+################################################################################
+
+FROM python:3
+
+WORKDIR /usr/src/app
+
+COPY . .
+
+RUN pip install --no-cache-dir -r requirements.txt
+
+
+CMD python ./syncjira.py
diff --git a/.test-infra/metrics/sync/jira/README.md
b/.test-infra/metrics/sync/jira/README.md
new file mode 100644
index 00000000000..af396f1f0bb
--- /dev/null
+++ b/.test-infra/metrics/sync/jira/README.md
@@ -0,0 +1,27 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+# Running script locally
+You can either run it via docker-compose that is located in
.test-infra/metrics/
+or run manually via docker. Below is insturctions for running script with
+docker.
+
+1. Build container `docker build -t syncjenkins .`
+2. Edit script to initialize db host via subprocess.
+3. `docker run -it --rm --name sync -v "$PWD":/usr/src/myapp -w /usr/src/myapp
-e "DB_PORT=5432" -e "DB_DBNAME=beam_metrics" -e "DB_DBUSERNAME=admin" -e
"DB_DBPWD=<password>" syncjira python syncjira.py`
+
diff --git a/.test-infra/metrics/sync/jira/requirements.txt
b/.test-infra/metrics/sync/jira/requirements.txt
new file mode 100644
index 00000000000..2130f52af5a
--- /dev/null
+++ b/.test-infra/metrics/sync/jira/requirements.txt
@@ -0,0 +1,18 @@
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+requests
+psycopg2-binary
+
diff --git a/.test-infra/metrics/sync/jira/syncjira.py
b/.test-infra/metrics/sync/jira/syncjira.py
new file mode 100644
index 00000000000..4ce3cac75c1
--- /dev/null
+++ b/.test-infra/metrics/sync/jira/syncjira.py
@@ -0,0 +1,242 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+
+# Queries Jira to collect metrics and put them in postgresql.
+import os
+import psycopg2
+import re
+import requests
+import socket
+import sys
+import time
+
+from datetime import datetime, timedelta
+from xml.etree import ElementTree
+
+
+# Keeping this as reference for localhost debug
+# Fetching docker host machine ip for testing purposes.
+# Actual host should be used for production.
+# import subprocess
+# cmd_out = subprocess.check_output(["ip", "route", "show"]).decode("utf-8")
+# host = cmd_out.split(" ")[2]
+
+host = os.environ['DB_HOST']
+port = os.environ['DB_PORT']
+dbname = os.environ['DB_DBNAME']
+dbusername = os.environ['DB_DBUSERNAME']
+dbpassword = os.environ['DB_DBPWD']
+
+jiraIssuesTableName = 'jira_issues'
+
+jiraIssuesCreateTableQuery = f"""
+create table {jiraIssuesTableName} (
+id integer PRIMARY KEY,
+key varchar,
+creator varchar,
+assignee varchar,
+status varchar,
+labels varchar,
+summary varchar,
+created TIMESTAMP,
+resolutiondate TIMESTAMP)
+"""
+
+jiraIssuesMetadataTableName = 'jira_issues_metadata'
+
+jiraIssuesMetadataCreateTableQuery = f"""
+create table {jiraIssuesMetadataTableName} (
+lastsynctime TIMESTAMP
+)
+"""
+
+def fetchIssues(startTime, startAt = 0):
+ startTimeStr = startTime.strftime("%Y-%m-%d %H:%M")
+ # time format 0001-01-01 00:01 ==> "yyyy-mm-dd hh:mm"
+ url = (f'https://issues.apache.org/jira/rest/api/latest/search?'
+ f'jql=project=BEAM '
+ f'AND updated > "{startTimeStr} "'
+ f'AND component=test-failures'
+ f'&maxResults=100&startAt={startAt}')
+ print(url)
+
+ r = requests.get(url)
+ return r.json()
+
+
+def initDBConnection():
+ conn = psycopg2.connect(f"dbname='{dbname}' user='{dbusername}'
host='{host}'"
+ f" port='{port}' password='{dbpassword}'")
+ return conn
+
+
+def tableExists(cursor, tableName):
+ cursor.execute(f"select * from information_schema.tables"
+ f" where table_name='{tableName}';")
+ return bool(cursor.rowcount)
+
+
+def initDBTablesIfNeeded():
+ connection = initDBConnection()
+ cursor = connection.cursor()
+
+ buildsTableExists = tableExists(cursor, jiraIssuesTableName)
+ print('Builds table exists', buildsTableExists)
+ if not buildsTableExists:
+ cursor.execute(jiraIssuesCreateTableQuery)
+ if not bool(cursor.rowcount):
+ raise Exception(f"Failed to create table {jiraIssuesTableName}")
+
+ metadataTableExists = tableExists(cursor, jiraIssuesMetadataTableName)
+ print('Metadata table exists', buildsTableExists)
+ if not metadataTableExists:
+ cursor.execute(jiraIssuesMetadataCreateTableQuery)
+ if not bool(cursor.rowcount):
+ raise Exception(f"Failed to create table {jiraIssuesMetadataTableName}")
+
+ minTimestamp = datetime(1970, 1, 1)
+ insertDummyTimestampSqlQuery = f"insert into {jiraIssuesMetadataTableName}
values (%s)"
+
+ cursor.execute(insertDummyTimestampSqlQuery, [minTimestamp])
+
+ cursor.close()
+ connection.commit()
+
+ connection.close()
+
+
+def updateLastSyncTimestamp(timestamp):
+ connection = initDBConnection()
+ cursor = connection.cursor()
+
+ cleanupQuery = f"delete from {jiraIssuesMetadataTableName}"
+ cursor.execute(cleanupQuery)
+ insertTimestampSqlQuery = f"insert into {jiraIssuesMetadataTableName} values
(%s)"
+ cursor.execute(insertTimestampSqlQuery, [timestamp])
+
+ cursor.close()
+ connection.commit()
+ connection.close()
+
+
+def fetchLastSyncTime():
+ connection = initDBConnection()
+ cursor = connection.cursor()
+
+ fetchQuery = f'select lastsynctime from {jiraIssuesMetadataTableName}'
+
+ cursor.execute(fetchQuery)
+ result = cursor.fetchone()[0]
+
+ cursor.close()
+ connection.close()
+ return result
+
+
+def buildRowValuesArray(issue):
+ fields = issue['fields']
+ values = [issue['id'],
+ issue['key'],
+ fields['creator']['name'],
+ fields['assignee']['name'] if fields['assignee'] is not None else
None,
+ fields['status']['name'],
+ fields['labels'],
+ fields['summary'],
+ fields['created'],
+ fields['resolutiondate']
+ ]
+ return values
+
+
+def insertRow(cursor, rowValues):
+ print(len(rowValues))
+ insertClause = (f'''insert into {jiraIssuesTableName}
+ values (%s, %s, %s, %s, %s, %s, %s, %s, %s)
+ ON CONFLICT (id) DO UPDATE
+ set
+ id = excluded.id,
+ key = excluded.key,
+ creator = excluded.creator,
+ assignee = excluded.assignee,
+ status = excluded.status,
+ labels = excluded.labels,
+ summary = excluded.summary,
+ created = excluded.created,
+ resolutiondate = excluded.resolutiondate'''
+ )
+
+ cursor.execute(insertClause, rowValues)
+
+
+def fetchNewData():
+ currentTimestamp = datetime.now()
+ lastSyncTimestamp = fetchLastSyncTime()
+
+ startAt = 0
+ total = 1
+
+ while (startAt < total):
+ queryResult = fetchIssues(lastSyncTimestamp, startAt)
+
+ newIssues = queryResult['issues']
+ fetchedCount = len(newIssues)
+
+ startAt += fetchedCount
+ total = queryResult['total']
+
+ connection = initDBConnection()
+ cursor = connection.cursor()
+
+ for issue in newIssues:
+ rowValues = buildRowValuesArray(issue)
+ insertRow(cursor, rowValues)
+
+ cursor.close()
+ connection.commit()
+ connection.close()
+
+ updateLastSyncTimestamp(currentTimestamp)
+ lastSyncTimestamp = fetchLastSyncTime()
+
+
+def probeJiraIsUp():
+ sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
+
+ # This is close enough for Jira
+ result = sock.connect_ex(('issues.apache.org', 443))
+ return True if result == 0 else False
+
+
+################################################################################
+if __name__ == '__main__':
+ print("Started.")
+
+ print("Checking if DB needs to be initialized.")
+ sys.stdout.flush()
+ initDBTablesIfNeeded()
+
+ print("Start jobs fetching loop.")
+ sys.stdout.flush()
+
+ while True:
+ if not probeJiraIsUp():
+ print("Jira is unavailable, skipping fetching data.")
+ continue
+ else:
+ print("Start fetching data.")
+ fetchNewData()
+ print("Done fetching data.")
+ print("Sleeping for 5 min.")
+ sys.stdout.flush()
+ time.sleep(5 * 60)
+
+ print('Done.')
+
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 155595)
Time Spent: 6h (was: 5h 50m)
> Create post-commit tests dashboard
> ----------------------------------
>
> Key: BEAM-5240
> URL: https://issues.apache.org/jira/browse/BEAM-5240
> Project: Beam
> Issue Type: Sub-task
> Components: testing
> Reporter: Mikhail Gryzykhin
> Assignee: Mikhail Gryzykhin
> Priority: Major
> Time Spent: 6h
> Remaining Estimate: 0h
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
