Hi Zhijie, Thanks for letting us know this issue has been recognized.
Thanks, Geoff On Mar 10, 2014, at 12:09 PM, Zhijie Shen <[email protected]> wrote: > Hi Geoff, > > Unfortunately, there's no such a API for users to determine whether the log > aggregation is completed or not, but the issue has been tackled. You can keep > an eye on YARN-1279. > > - Zhijie > > > On Mon, Mar 10, 2014 at 10:18 AM, Geoff Thompson <[email protected]> wrote: > Hello, > > Log aggregation is great. However, if a yarn application runs a large number > of tasks which generate large logs, it takes some finite amount of time for > all of the logs to be collected and written to the HDFS. > > Currently our client code runs the equivalent of the "yarn logs" command once > all tasks have completed. This works fine provided log aggregation is > complete. > > But it fails in a variety of ways if aggregation is not complete. This > includes one case where the "yarn logs" code encounters no exceptions and no > non-zero return codes from methods, but returns an empty string. > > So, is there a way to determine if log aggregation is complete? > > Thanks, > > Geoff > > > > > -- > Zhijie Shen > Hortonworks Inc. > http://hortonworks.com/ > > CONFIDENTIALITY NOTICE > NOTICE: This message is intended for the use of the individual or entity to > which it is addressed and may contain information that is confidential, > privileged and exempt from disclosure under applicable law. If the reader of > this message is not the intended recipient, you are hereby notified that any > printing, copying, dissemination, distribution, disclosure or forwarding of > this communication is strictly prohibited. If you have received this > communication in error, please contact the sender immediately and delete it > from your system. Thank You.
