Hi, Rui. Thank you for initiating this proposal.
I have a question regarding redirecting stdout and stderr to LOG: Will they be written to the taskManager.log file by default or the taskManager.out file? If we can make taskmanager.out splittable and rolling, would it be easier for users to use this feature? Best, Feng On Thu, Nov 9, 2023 at 3:15 AM Archit Goyal <argo...@linkedin.com.invalid> wrote: > Hi Rui, > > Thanks for the proposal. > > The proposed solution of supporting System out and err to be redirected to > LOG or discarded and introducing an enum and two options to manage this, > seems reasonable. > > +1 > > Thanks, > Archit Goyal > > > From: Piotr Nowojski <pnowoj...@apache.org> > Date: Wednesday, November 8, 2023 at 7:38 AM > To: dev@flink.apache.org <dev@flink.apache.org> > Subject: Re: [DISCUSS] FLIP-390: Support System out and err to be > redirected to LOG or discarded > Hi Rui, > > Thanks for the proposal. > > +1 I don't have any major comments :) > > One nit. In `SystemOutRedirectToLog` in this code: > > System.arraycopy(buf, count - LINE_SEPARATOR_LENGTH, bytes, 0, > LINE_SEPARATOR_LENGTH); > return Arrays.equals(LINE_SEPARATOR_BYTES, bytes) > > Is there a reason why you are suggesting to copy out bytes from `buf` to > `bytes`, > instead of using `Arrays.equals(int[] a, int aFromIndex, int aToIndex, > int[] b, int bFromIndex, int bToIndex)`? > > Best, > Piotrek > > śr., 8 lis 2023 o 11:53 Rui Fan <1996fan...@gmail.com> napisał(a): > > > Hi all! > > > > I would like to start a discussion of FLIP-390: Support System out and > err > > to be redirected to LOG or discarded[1]. > > > > In various production environments, either cloud native or physical > > machines, the disk space that Flink TaskManager can use is limited. > > > > In general, the flink users shouldn't use the `System.out.println` in > > production, > > however this may happen when the number of Flink jobs and job developers > > is very large. Flink job may use System.out to output a large amount of > > data > > to the taskmanager.out file. This file will not roll, it will always > > increment. > > Eventually the upper limit of what the TM can be used for is reached. > > > > We can support System out and err to be redirected to LOG or discarded, > > the LOG can roll and won't increment forever. > > > > This feature is useful for SREs who maintain Flink environments, they can > > redirect System.out to LOG by default. Although the cause of this problem > > is > > that the user's code is not standardized, for SRE, pushing users to > modify > > the code one by one is usually a very time-consuming operation. It's also > > useful for job stability where System.out is accidentally used. > > > > Looking forward to your feedback, thanks~ > > > > [1] > https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fx%2F4guZE&data=05%7C01%7Cargoyal%40linkedin.com%7C937821de7bd846e6b97408dbe070beae%7C72f988bf86f141af91ab2d7cd011db47%7C0%7C0%7C638350547072823674%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=zEv6B0Xiq2SNuU6Fm%2BAXnH%2BRILbm6Q0uDRbN7h6iaPM%3D&reserved=0 > <https://cwiki.apache.org/confluence/x/4guZE> > > > > Best, > > Rui > > >
