Hi vino Yes, it's only from the perspective of performance of reading log or metrics. If the logs with timestamps(e.g. jobmanager.log.2018-07-29) will never change, maybe blob store can cache some of them to improve performance.
BTW, please considering to develop an API for reading logs. I think many flink users meet this problem. Thanks! Xinyu Zhang 2018年7月30日星期一，vino yang <[email protected]> 写道： > Hi Xinyu, > > Thank you for your clarification on "periodic reading". If Flink considers > developing an API for reading logs, I think this is a good idea. > > Regarding the problem of TM reading logs, your idea is good from a > performance perspective. > But Flink didn't provide any web services for the TM from the beginning. > All the requests were passed through the JM proxy. > Just because of the log read performance changes, there will be major > changes to the architecture, and this will make the TM take on more > responsibilities. > > Thanks, vino. > > > 2018-07-30 17:34 GMT+08:00 Xinyu Zhang <[email protected]>: > >> Thanks for your reply. "periodic reading" means reading all logs in a >> given time interval. For example, my logs is daily divided, I can get all >> logs of yesterday through a parameter like '2018-07-29/2018-07-30'. >> >> TM which provides a web service to display information will lessen the >> burden of jobmanager, especially when there are many taskManagers in the >> flink cluster. >> >> >> 2018年7月30日星期一，vino yang <[email protected]> 写道： >> >>> Hi Xinyu, >>> >>> This is indeed a problem. Especially when the amount of logs is large, >>> it may even cause the UI to stall for a long time. The same is true for >>> YRAN, and there is really no good way to do it at the moment. >>> Thank you for your suggestion, do you mean "periodic reading" refers to >>> full or incremental? If it is a full reading, I personally do not recommend >>> this, which will increase the burden on JM and client, and it is >>> appropriate to manually trigger it by the user. Maybe we can consider >>> loading only a few logs at a time, such as incremental reads, paged >>> displays, and so on. >>> Regarding the second question, Flink did this because its TM does not >>> provide any web services to display information, and the Web UI is >>> currently bundled with JM. >>> >>> Thanks, vino. >>> >>> 2018-07-30 16:33 GMT+08:00 Xinyu Zhang <[email protected]>: >>> >>>> Hi all >>>> >>>> We use flink on yarn and flink version is 1.4. >>>> >>>> When a streaming job run for a long time, the webUI cannot show logs. >>>> This may be becasue the log size is too large. >>>> >>>> However, if we use the DailyRollingAppender to divide logs (granularity >>>> is `day`) in log4j.properties, we will never see the log of yesterday. >>>> >>>> Is there any ideas can make read logs easier? >>>> >>>> Maybe, we should add an interface that support for reading log by time >>>> interval. Besides, when we get the taskmanager logs through webUI, >>>> jobmanager can redirect to a URL of the taskmanager, which users can get >>>> the logs directly (Just like MR task), other than downloading the logs from >>>> taskmanager and then sending logs to users. >>>> >>>> Thanks! >>>> >>>> Xinyu Zhang >>>> >>>> >>>> >>> >
