----- Original Message ----- > From: "Matthew Khouzam" <[email protected]> > To: [email protected], "Linux Tools developer discussions" > <[email protected]> > Sent: Tuesday, November 26, 2013 11:44:46 AM > Subject: [lttng-dev] Tmf Ctf Parser Scalability issues (1/3) > > Hello all, > I was looking at the CTF parser in the TMF project of Linux Tools. I > have come up with three points where scalability will be an issue. I am > sending 3 emails, each one describing one of the issues so we can > aggregate them more cohesively. First a primer, CTF is a file format > that has traces written in packets, the packets are parts of files in > streams, the streams are files in a directory. > > Issue 1: Number of open files > CTF can have an unlimited amount of stream files. The parser opens all > of them simultaneously. This means if you have say over 1024 or whatever > your file limit is, files opened simultaneously, you will hit a wall. An > example of this would be the new Xeon Phi boards that have something > like 255 cores, if you have more than 4 channels per core, you're going > to hit this issue. > > Proposed solution: > I envision fixing them by setting up a pool of files and when the limit > is hit, swapping files in and out, incurring a performance hit but still > allowing the program to run. Any objections?
No objection. I would think a LRU list of most recently used files would be a good data structure for this. I already described this in a bug against babeltrace: http://bugs.lttng.org/issues/649 Thanks! Mathieu > > _______________________________________________ > lttng-dev mailing list > [email protected] > http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev > -- Mathieu Desnoyers EfficiOS Inc. http://www.efficios.com _______________________________________________ lttng-dev mailing list [email protected] http://lists.lttng.org/cgi-bin/mailman/listinfo/lttng-dev
