[jira] [Commented] (YARN-5915) ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every event write
[ https://issues.apache.org/jira/browse/YARN-5915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15715721#comment-15715721 ] Hudson commented on YARN-5915: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10928 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/10928/]) YARN-5915. ATS 1.5 FileSystemTimelineWriter causes flush() to be called (jlowe: rev f304ccae3c2e0849b0b0b24c4bfe7a3a1ec2bb94) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/FileSystemTimelineWriter.java > ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every > event write > > > Key: YARN-5915 > URL: https://issues.apache.org/jira/browse/YARN-5915 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 3.0.0-alpha1 >Reporter: Atul Sikaria >Assignee: Atul Sikaria > Fix For: 2.8.0, 3.0.0-alpha2 > > Attachments: YARN-5915.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5915) ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every event write
[ https://issues.apache.org/jira/browse/YARN-5915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15713331#comment-15713331 ] Junping Du commented on YARN-5915: -- Cool. +1 as well. > ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every > event write > > > Key: YARN-5915 > URL: https://issues.apache.org/jira/browse/YARN-5915 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 3.0.0-alpha1 >Reporter: Atul Sikaria >Assignee: Atul Sikaria > Attachments: YARN-5915.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5915) ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every event write
[ https://issues.apache.org/jira/browse/YARN-5915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15713282#comment-15713282 ] Jason Lowe commented on YARN-5915: -- Sure if the output stream could be unbuffered then having it buffer at the json generator is fine, and that's what the patch does. +1 for the patch, I'll commit tomorrow if no objections. > ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every > event write > > > Key: YARN-5915 > URL: https://issues.apache.org/jira/browse/YARN-5915 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 3.0.0-alpha1 >Reporter: Atul Sikaria >Assignee: Atul Sikaria > Attachments: YARN-5915.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5915) ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every event write
[
https://issues.apache.org/jira/browse/YARN-5915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15709863#comment-15709863
]
Junping Du commented on YARN-5915:
--
bq. Is the output stream unbuffered such that the write is acting like a flush?
That's also what I am suspecting. Shall we assume the behavior of buffered or
unbuffered output stream be consistent in our case? The _flushBuffer() is hard
to do so. May be good to buffer things at JsonGenerator?
{noformat}
protected final void _flushBuffer() throws IOException
{
int len = _outputTail - _outputHead;
if (len > 0) {
int offset = _outputHead;
_outputTail = _outputHead = 0;
_writer.write(_outputBuffer, offset, len);
}
}
{noformat}
> ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every
> event write
>
>
> Key: YARN-5915
> URL: https://issues.apache.org/jira/browse/YARN-5915
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 3.0.0-alpha1
>Reporter: Atul Sikaria
>Assignee: Atul Sikaria
> Attachments: YARN-5915.01.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5915) ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every event write
[
https://issues.apache.org/jira/browse/YARN-5915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15683683#comment-15683683
]
Jason Lowe commented on YARN-5915:
--
Sorry for missing this in YARN-4814. I thought the fix was equivalent because
calling flush on the json generator won't actually flush the underlying stream
if JsonGenerator.Feature.FLUSH_PASSED_TO_STREAM is set to false:
{code}
@Override
public final void flush()
throws IOException
{
_flushBuffer();
if (_outputStream != null) {
if (isEnabled(Feature.FLUSH_PASSED_TO_STREAM)) {
_outputStream.flush();
}
}
}
{code}
Could you elaborate on how the filesystem flush is getting called? The mapper
code is calling jgen.flush on every write, but how is that making it to the
filesystem if FLUSH_PASSED_TO_STREAM is false on the json generator? If that
property is false it appears all flushing the json generator will do is simply
writing to the underlying output stream (but not flushing), which seems like
exactly what we want. Is the output stream unbuffered such that the write is
acting like a flush?
> ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every
> event write
>
>
> Key: YARN-5915
> URL: https://issues.apache.org/jira/browse/YARN-5915
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 3.0.0-alpha1
>Reporter: Atul Sikaria
>Assignee: Atul Sikaria
> Attachments: YARN-5915.01.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5915) ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every event write
[ https://issues.apache.org/jira/browse/YARN-5915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15680063#comment-15680063 ] Junping Du commented on YARN-5915: -- Also, CC [~gtCarrera9] as well. > ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every > event write > > > Key: YARN-5915 > URL: https://issues.apache.org/jira/browse/YARN-5915 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 3.0.0-alpha1 >Reporter: Atul Sikaria >Assignee: Atul Sikaria > Attachments: YARN-5915.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5915) ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every event write
[ https://issues.apache.org/jira/browse/YARN-5915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15680062#comment-15680062 ] Junping Du commented on YARN-5915: -- Thanks [~ASikaria] for reporting the issue and delivering a fix. I have added you to yarn contributor list, and assign this jira to you. The fix sounds reasonable to me. CC [~xgong] and [~jlowe] for more comments who worked on YARN-4814 before. > ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every > event write > > > Key: YARN-5915 > URL: https://issues.apache.org/jira/browse/YARN-5915 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 3.0.0-alpha1 >Reporter: Atul Sikaria >Assignee: Atul Sikaria > Attachments: YARN-5915.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5915) ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every event write
[
https://issues.apache.org/jira/browse/YARN-5915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15678534#comment-15678534
]
Hadoop QA commented on YARN-5915:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
54s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m
16s{color} | {color:green} hadoop-yarn-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
15s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 17m 50s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:a9ad5d6 |
| JIRA Issue | YARN-5915 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12839675/YARN-5915.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 59dc9cc3a3da 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12
17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 7584fbf |
| Default Java | 1.8.0_111 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-YARN-Build/13977/testReport/ |
| modules | C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common U:
hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/13977/console |
| Powered by | Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every
> event write
>
>
> Key: YARN-5915
> URL: https://issues.apache.org/jira/browse/YARN-5915
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 3.0.0-alpha1
>Reporter: Atul Sikaria
> At
[jira] [Commented] (YARN-5915) ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every event write
[ https://issues.apache.org/jira/browse/YARN-5915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15678494#comment-15678494 ] Atul Sikaria commented on YARN-5915: Attached patch that should address this issue. > ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every > event write > > > Key: YARN-5915 > URL: https://issues.apache.org/jira/browse/YARN-5915 > Project: Hadoop YARN > Issue Type: Bug > Components: timelineserver >Affects Versions: 3.0.0-alpha1 >Reporter: Atul Sikaria > Attachments: YARN-5915.01.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-5915) ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every event write
[
https://issues.apache.org/jira/browse/YARN-5915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15678475#comment-15678475
]
Atul Sikaria commented on YARN-5915:
This was seen previously as well, in YARN-4814.
The issue is with writeEntities method in FileSystemTimelineWriter
(https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/client/api/impl/FileSystemTimelineWriter.java#L317).
This calls getObjectMapper().writeValue(…), which does a flush() after every
write with default config.
{noformat}
@Override
public void writeValue(JsonGenerator jgen, Object value)
throws IOException, JsonGenerationException, JsonMappingException
{
SerializationConfig config = copySerializationConfig();
if (config.isEnabled(SerializationConfig.Feature.CLOSE_CLOSEABLE) && (value
instanceof Closeable)) {
_writeCloseableValue(jgen, value, config);
} else {
_serializerProvider.serializeValue(config, jgen, value,
_serializerFactory);
if
(config.isEnabled(SerializationConfig.Feature.FLUSH_AFTER_WRITE_VALUE)) {
jgen.flush();
}
}
}
{noformat}
On filesystems that map flush() to no-op or trivial operations, this is not a
big deal. But on filesystems where flush() incurs a larger cost, this becomes a
bottleneck for timeline events flow.
> ATS 1.5 FileSystemTimelineWriter causes flush() to be called after every
> event write
>
>
> Key: YARN-5915
> URL: https://issues.apache.org/jira/browse/YARN-5915
> Project: Hadoop YARN
> Issue Type: Bug
> Components: timelineserver
>Affects Versions: 3.0.0-alpha1
>Reporter: Atul Sikaria
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
