[
https://issues.apache.org/jira/browse/GOBBLIN-1386?focusedWorklogId=551093&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-551093
]
ASF GitHub Bot logged work on GOBBLIN-1386:
-------------------------------------------
Author: ASF GitHub Bot
Created on: 11/Feb/21 01:48
Start Date: 11/Feb/21 01:48
Worklog Time Spent: 10m
Work Description: sv2000 commented on a change in pull request #3226:
URL: https://github.com/apache/gobblin/pull/3226#discussion_r574206234
##########
File path:
gobblin-core/src/main/java/org/apache/gobblin/writer/PartitionedDataWriter.java
##########
@@ -130,7 +130,8 @@ public PartitionedDataWriter(DataWriterBuilder<S, D>
builder, final State state)
this.state.setProp(WRITER_LATEST_SCHEMA, builder.getSchema());
}
long cacheExpiryInterval =
this.state.getPropAsLong(PARTITIONED_WRITER_CACHE_TTL_SECONDS,
DEFAULT_PARTITIONED_WRITER_CACHE_TTL_SECONDS);
- this.writeTimeoutInterval = cacheExpiryInterval / 3;
+ // Increase the timeout value to make it less sensitive to HDFS slow writer
+ this.writeTimeoutInterval = cacheExpiryInterval / 3 * 2;
Review comment:
Do we want to derive this from Cache TTL value or can this be an
independent config? The reason being the right value for write timeout may vary
depending on the destination being written to. And if the config exceeds the
cache ttl, we can bound the value to the TTL.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 551093)
Time Spent: 20m (was: 10m)
> Increase the timeout setting for Gobblin Partitioned DataWriter to write a
> record
> ---------------------------------------------------------------------------------
>
> Key: GOBBLIN-1386
> URL: https://issues.apache.org/jira/browse/GOBBLIN-1386
> Project: Apache Gobblin
> Issue Type: Task
> Reporter: Zihan Li
> Priority: Major
> Time Spent: 20m
> Remaining Estimate: 0h
>
> Increase the timeout setting for Gobblin Partitioned DataWriter to write a
> record to make it less sensitive to the slow HDFS writer
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
