[jira] [Updated] (TEZ-4028) Events not visible from proto history logging for s3a filesystem until dag completes.
[ https://issues.apache.org/jira/browse/TEZ-4028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Eagles updated TEZ-4028: - Fix Version/s: 0.9.2 > Events not visible from proto history logging for s3a filesystem until dag > completes. > - > > Key: TEZ-4028 > URL: https://issues.apache.org/jira/browse/TEZ-4028 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Labels: history > Fix For: 0.9.2, 0.10.1 > > Attachments: TEZ-4028.01.patch, TEZ-4028.02.patch > > > The events are not visible in the files because s3 filesystem > * flush writes to local disk and only upload/commit to s3 on close. > * does not support append > As an initial fix we log the dag submitted, initialized and started events > into a file and these can be read to get the dag plan, config from the AM. > The counters are anyways not available until the dag completes. > The in-progress information cannot be read, this can be obtained from the AM > once we have the above events. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (TEZ-4028) Events not visible from proto history logging for s3a filesystem until dag completes.
[ https://issues.apache.org/jira/browse/TEZ-4028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16783801#comment-16783801 ] Jonathan Eagles commented on TEZ-4028: -- Cherry-picked this change to branch-0.9 line as well. > Events not visible from proto history logging for s3a filesystem until dag > completes. > - > > Key: TEZ-4028 > URL: https://issues.apache.org/jira/browse/TEZ-4028 > Project: Apache Tez > Issue Type: Bug >Reporter: Harish Jaiprakash >Assignee: Harish Jaiprakash >Priority: Major > Labels: history > Fix For: 0.9.2, 0.10.1 > > Attachments: TEZ-4028.01.patch, TEZ-4028.02.patch > > > The events are not visible in the files because s3 filesystem > * flush writes to local disk and only upload/commit to s3 on close. > * does not support append > As an initial fix we log the dag submitted, initialized and started events > into a file and these can be read to get the dag plan, config from the AM. > The counters are anyways not available until the dag completes. > The in-progress information cannot be read, this can be obtained from the AM > once we have the above events. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
