Thanks Lee!
Yes I posted a request in gobblin forum waiting for some advise.
Here one thing puzzling me is, if I check the log each task of the job has
debug logs suggesting it changed state from RUNNING to COMPLETED at some point
but job is still unfinished.
The capacity full message was for task runner 2. But task _3 was first assigned
to task runner 1 and then moved to task runner 4 and transitioned from running
to completed but still no result file.
I will try increasing capacity from 40/ increase number of task runners but I
didn’t see any time out/abort logs.
“ArchiveJob16_1577133620749_3 has completed with state COMPLETED. Marking as
such in rebalancer context.”
But then no log suggesting any task is pending in RUNNING. But no result file
in task output dir.
And then logs like
2019-12-23 21:01:02 UTC DEBUG [GenericHelixController-event_process]
org.apache.helix.task.JobRebalancer - All partitions: [0, 1, 2, 3, 4, 5, 6, 7,
8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27,
28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47,
48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62] taskAssignment:
{GobblinYarnTaskRunner_1=[], GobblinYarnTaskRunner_2=[],
GobblinYarnTaskRunner_3=[], GobblinYarnTaskRunner_4=[]} excludedInstances: []
2019-12-23 21:01:02 UTC WARN [GenericHelixController-event_process]
org.apache.helix.task.assigner.ThreadCountBasedTaskAssigner - No task to
assign!
2019-12-23 21:01:02 UTC DEBUG [GenericHelixController-event_process]
org.apache.helix.manager.zk.zookeeper.ZkClient - Waiting for keeper state
SyncConnected
2019-12-23 21:01:02 UTC DEBUG [GenericHelixController-event_process]
org.apache.helix.manager.zk.zookeeper.ZkClient - State is SyncConnected
Thanks & Regards,
Praveen
From: Hunter Lee <[email protected]>
Sent: Friday, December 27, 2019 4:30 PM
To: [email protected]
Subject: Re: apache gobblin on yarn using helix - job getting stuck without
error message
[THIS MESSAGE ORIGINATED FROM A NON-IGT EMAIL ADDRESS]
2019-12-23 20:40:28 UTC WARN [GenericHelixController-event_process]
org.apache.helix.task.assigner.AssignableInstance - AssignableInstance does
not have enough capacity for quotaType: DEFAULT. Task:
23b17106-6e18-4516-9745-879a3f6a30b8, quotaType: DEFAULT, Instance name:
GobblinYarnTaskRunner_2. Current capacity: 40 capacity needed to schedule: 40
The log message above is telling. Helix Task Framework has a statically defined
capacity per instance of 40 tasks. For some reason, you have all of your
capacity full for GobblinYarnTaskRunner_2 (which should be a Helix Participant).
This may mean things like:
1. You should add more Helix Participants (task runners) to give the cluster
more capacity for tasks.
2. Your tasks are not completing properly - meaning they are stuck in RUNNING
state. This usually is due to the user-defined logic (either your own or Apache
Gobblin's task ingestion logic).
We collaborate with Apache Gobblin quite often and they have been using Helix
Task Framework for data ingestion with success. I also suggest you reach out to
Gobblin's community to see if your configs are set correctly.
Hunter
On Fri, Dec 27, 2019 at 12:30 PM Krishnan Nair, Praveen
<[email protected]<mailto:[email protected]>> wrote:
Hi,
I am trying to configure apache gobblin on yarn to pull data from postgres to
hdfs to store as avro files in daily partitions. Gobblin uses helix version :
0.8.2 to manage the tasks.
I am facing an issue as the job gets stuck when data volume is increased with
some of the tasks getting completed(as per debug logs) but result files are
missing.
There are 63 tasks one for each partition for this job and I can see from logs
4 task runners are initialized and assigned tasks.
After creating most of the task result files in task output dir, the job is
getting stuck – with no error message/exception.
For a reduced volume of data same configuration works and the job finishes. If
it is getting stuck that happens roughly in 25 – 30 mins.
One of such tasks with log as COMPLETED but file missing in output dir is
…_1577133620749_3 as shown below.
2019-12-23 20:40:28 UTC WARN [GenericHelixController-event_process]
org.apache.helix.task.assigner.AssignableInstance - AssignableInstance does
not have enough capacity for quotaType: DEFAULT. Task:
23b17106-6e18-4516-9745-879a3f6a30b8, quotaType: DEFAULT, Instance name:
GobblinYarnTaskRunner_2. Current capacity: 40 capacity needed to schedule: 40
2019-12-23 20:40:32 UTC DEBUG [GenericHelixController-event_process]
org.apache.helix.task.AbstractTaskDispatcher - Setting task partition
job_ArchiveJob16_1577133620749_job_ArchiveJob16_1577133620749_3 state to
RUNNING on instance GobblinYarnTaskRunner_1
2019-12-23 20:48:38 UTC DEBUG [GenericHelixController-event_process]
org.apache.helix.task.AbstractTaskDispatcher - Task partition
job_ArchiveJob16_1577133620749_job_ArchiveJob16_1577133620749_3 has a pending
state transition on instance GobblinYarnTaskRunner_4. Using the previous ideal
state which was RUNNING.
2019-12-23 20:50:20 UTC DEBUG [GenericHelixController-event_process]
org.apache.helix.task.AbstractTaskDispatcher - Setting task partition
job_ArchiveJob16_1577133620749_job_ArchiveJob16_1577133620749_3 state to
RUNNING on instance GobblinYarnTaskRunner_4.
2019-12-23 20:50:22 UTC DEBUG [GenericHelixController-event_process]
org.apache.helix.task.AbstractTaskDispatcher - Task partition
job_ArchiveJob16_1577133620749_job_ArchiveJob16_1577133620749_3 has completed
with state COMPLETED. Marking as such in rebalancer context.
Not sure how to figure out what is happening with the job. Appreciate any
advice/suggestions.
Thanks & Regards,
Praveen
CONFIDENTIALITY NOTICE: This message is the property of International Game
Technology PLC and/or its subsidiaries and may contain proprietary,
confidential or trade secret information. This message is intended solely for
the use of the addressee. If you are not the intended recipient and have
received this message in error, please delete this message from your system.
Any unauthorized reading, distribution, copying, or other use of this message
or its attachments is strictly prohibited.
