I have found a performance issue when using TextIO.StreamIO.input1 to
read a functional stream. Looking at gc/non-gc times and using
PolyML.profiling, it appears that garbage collection accounts for most
of the time. There is some code below to demonstrate with stats that
include comparison with SML/NJ.
The profiling shows that readFromReader in basis/BasicStreamIO.sml is
responsible for creating values that are being garbage collected.
Looking at this code, I can see various things that would contribute to
this garbage collection but nothing that is obviously problematic. Is
it simply the case that overheads in the implementation mean that it is
not suitable for a large number of small reads?
Thanks,
Phil
(* `makeTextFile (filename, length)` generates a text file *)
(* called `filename` of length `length` where every character *)
(* is the digit '0'. *)
local
val l = 2048;
val s = implode (List.tabulate (l, fn _ => #"0"));
in
fun makeTextFile (filename, length) =
let
open TextIO;
val ostream = getOutstream (openOut filename);
fun write n =
if n < l
then
StreamIO.output (ostream, String.substring (s, 0, n))
else
(StreamIO.output (ostream, s); write (n - l));
in
write length;
StreamIO.closeOut ostream
end;
end;
(* `readTextFile filename` reads the file called `filename` *)
(* one character at a time using a functional input stream *)
(* and returns the time taken to read it. *)
fun readTextFile filename =
let
open TextIO;
val src = getInstream (openIn filename);
fun read src =
case StreamIO.input1 src of
SOME (_, src') => read src'
| NONE => ();
val startTime = Timer.startCPUTimer ();
in
read src;
(fn {sys, usr} => Time.+ (sys, usr)) (Timer.checkCPUTimer startTime)
before StreamIO.closeIn src
end;
(* Create files and then read *)
makeTextFile ("text-01MB", 1024 * 1024 * 1);
makeTextFile ("text-02MB", 1024 * 1024 * 2);
makeTextFile ("text-03MB", 1024 * 1024 * 3);
makeTextFile ("text-04MB", 1024 * 1024 * 4);
makeTextFile ("text-05MB", 1024 * 1024 * 5);
readTextFile "text-01MB";
readTextFile "text-02MB";
readTextFile "text-03MB";
readTextFile "text-04MB";
readTextFile "text-05MB";
(* read times in seconds on Intel Core 2 Duo CPU T7500 @ 2.20GHz, 2GB *)
(*
Poly/ML 4.1.3 Poly/ML 5.2 SML/NJ 110.52
text-01MB 0.7 - 0.8 0.3 - 1.5 0.03 - 0.05
text-02MB 2.2 - 2.3 1.9 - 2.8 0.06 - 0.09
text-03MB 4.3 - 4.5 2.0 - 4.3 0.08 - 0.14
text-04MB 7.2 - 7.5 4.2 - 6.6 0.11 - 0.16
text-05MB 11.0 - 11.1 4.0 - 5.8 0.14 - 0.18
*)
(* for large files, Poly/ML 4.1.3 can run out of store: *)
makeTextFile ("text-40MB", 1024 * 1024 * 40);
readTextFile "text-40MB";
(*
Poly/ML 4.1.3 Poly/ML 5.2 SML/NJ 110.52
text-40MB out of store 101.4 1.1 - 1.2
4.1.3 gives the following message:
Run out of store - interrupting console processes
Exception- Interrupt raised
*)
The information contained in this E-Mail and any subsequent
correspondence is private and is intended solely for the intended
recipient(s). The information in this communication may be
confidential and/or legally privileged. Nothing in this e-mail is
intended to conclude a contract on behalf of QinetiQ or make QinetiQ
subject to any other legally binding commitments, unless the e-mail
contains an express statement to the contrary or incorporates a formal Purchase Order.
For those other than the recipient any disclosure, copying,
distribution, or any action taken or omitted to be taken in reliance
on such information is prohibited and may be unlawful.
Emails and other electronic communication with QinetiQ may be
monitored and recorded for business purposes including security, audit
and archival purposes. Any response to this email indicates consent
to this.
Telephone calls to QinetiQ may be monitored or recorded for quality
control, security and other business purposes.
QinetiQ Limited
Registered in England & Wales: Company Number:3796233
Registered office: 85 Buckingham Gate, London SW1E 6PD, United Kingdom
Trading address: Cody Technology Park, Cody Building, Ively Road, Farnborough, Hampshire, GU14 0LX, United Kingdom
http://www.QinetiQ.com/home/legal.html
_______________________________________________
polyml mailing list
polyml@inf.ed.ac.uk
http://lists.inf.ed.ac.uk/mailman/listinfo/polyml
