[
https://issues.apache.org/jira/browse/GOBBLIN-1964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apekshit Kumar updated GOBBLIN-1964:
------------------------------------
Description:
*Context :*
Currently due to NN availability issues, acquire job lock is failing, because
of which job fails.
{code:java}
select deployment_id, status, count(*) from gobblin_job_queue where
created_date >= '2021-09-01' and created_date < '2021-10-01' and
failure_exception like '%NullPointerException%' group by deployment_id, status
order by deployment_id, status;
+---------------+--------+----------+
| deployment_id | status | count(*) |
+---------------+--------+----------+
| 1 | FAILED | 253 |
| 2 | FAILED | 6 |
| 230 | FAILED | 157 |
| 22702 | FAILED | 11 |
| 22703 | FAILED | 13 |
| 22704 | FAILED | 2 |
+---------------+--------+----------+
6 rows in set (1.04 sec)
mysql> select deployment_id, status, count(*) from gobblin_job_queue where
created_date >= '2021-08-01' and created_date < '2021-09-01' and
failure_exception like '%NullPointerException%' group by deployment_id, status
order by deployment_id, status;
+---------------+--------+----------+
| deployment_id | status | count(*) |
+---------------+--------+----------+
| 1 | FAILED | 1091 |
| 3 | FAILED | 1598 |
| 230 | FAILED | 15870 |
+---------------+--------+----------+
3 rows in set (1.18 sec)
{code}
*Acceptance Criteria:*
Job lock acquisition to be made resilient to NN issues, probably by moving
locks to Zk or retrying while acquiring lock, in case of NN issues
(IOExceptions)@
was:
*Context:*
Following a restart, Gobblin service is currently unable to process previous
jobs in the RUNNING/LAUNCHED/SUBMITTED state, resulting in a stuck state for
these jobs.
*Example scenario mentioned here*
A job is in the LAUNCHED state, and while calculating CDC, the Application
master got re-attempted, actually due to name node issue (can be any env
issues).
!2ac4a7d8-f168-4877-a583-03d62f1384ac.png|width=1179,height=574!
*As the job state in DB :*
{code:java}
mysql> select * from gobblin_job_queue where
job_name='DM-JOB-fpti-druid-dp-venmo' order by created_date desc limit 10;
+------------------------------------------+----------------------------+---------------+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+----------------------------------------------+---------------------+---------------------+
| queue_id | job_name | deployment_id | failure_exception | configs | status |
job_id | created_date | updated_date |
+------------------------------------------+----------------------------+---------------+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------+----------------------------------------------+---------------------+---------------------+
| DM-JOB-fpti-druid-dp-venmo_1630444318758 | DM-JOB-fpti-druid-dp-venmo | 2 |
NULL |
\{"dataset":{"batch_id":"20210831211155","name":"default._druid-test_dataproc-jobs_venmo","snapshot_id":"20210831211155"},"gobblin":\{"client":{"id":"AIRFLOW_PAZ_DMP_DO"},"deployment":\{"name":"DMP228"}},"namespace":"Chunnel"}
| LAUNCHED | job_DM-JOB-fpti-druid-dp-venmo_1630444325903 | 2021-08-31
21:12:00 | 2021-08-31 21:12:38 |
{code}
*Acceptance Criteria:*
# Gobblin Jobs should be resumed, even if GobblinAppMaster gets restarted when
the Jobs are not finalized.
# The system should automatically resume jobs that were in the
RUNNING/LAUNCHED/SUBMITTED state after the restart.
# The solution should address lingering locks acquired in the previous run.
# It should not pick up the jobs/clean locks that are being picked up by other
deployments, as part of work stealing.
> [Gobblin] Job lock acquisition should be made resilient
> -------------------------------------------------------
>
> Key: GOBBLIN-1964
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1964
> Project: Apache Gobblin
> Issue Type: Bug
> Components: misc
> Affects Versions: 0.15.0
> Reporter: Apekshit Kumar
> Priority: Minor
>
> *Context :*
> Currently due to NN availability issues, acquire job lock is failing, because
> of which job fails.
>
> {code:java}
> select deployment_id, status, count(*) from gobblin_job_queue where
> created_date >= '2021-09-01' and created_date < '2021-10-01' and
> failure_exception like '%NullPointerException%' group by deployment_id,
> status order by deployment_id, status;
> +---------------+--------+----------+
> | deployment_id | status | count(*) |
> +---------------+--------+----------+
> | 1 | FAILED | 253 |
> | 2 | FAILED | 6 |
> | 230 | FAILED | 157 |
> | 22702 | FAILED | 11 |
> | 22703 | FAILED | 13 |
> | 22704 | FAILED | 2 |
> +---------------+--------+----------+
> 6 rows in set (1.04 sec)
> mysql> select deployment_id, status, count(*) from gobblin_job_queue where
> created_date >= '2021-08-01' and created_date < '2021-09-01' and
> failure_exception like '%NullPointerException%' group by deployment_id,
> status order by deployment_id, status;
> +---------------+--------+----------+
> | deployment_id | status | count(*) |
> +---------------+--------+----------+
> | 1 | FAILED | 1091 |
> | 3 | FAILED | 1598 |
> | 230 | FAILED | 15870 |
> +---------------+--------+----------+
> 3 rows in set (1.18 sec)
> {code}
> *Acceptance Criteria:*
> Job lock acquisition to be made resilient to NN issues, probably by moving
> locks to Zk or retrying while acquiring lock, in case of NN issues
> (IOExceptions)@
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
