On 28/10/16 16:21, A. Soroka wrote:
Okay, that's pretty much the sensible answer I expected: "YMMV, try some experiments
and let's work with hard data."
I need to reread Andy's post [1] about the packing algorithm to make sure I
understand what's going on there.
The hardware in question is definitely "$$$"-- maybe even "$$$$". {grin} This
is contract work, so before going too much further I need to check with my principals to prioritize
this kind of speculative work and we will make a plan accordingly.
I'm not sure immediately how to measure the rates in terms of tuples/sec-- I'm
not aware of any such logging from sort itself, but I haven't looked for it. I
assume that tdbloader2 can emit messages as it packs tuples in the same way
that it emits messages with an average rate periodically in the 'data' phase?
It prints the elapsed time and the number of triples/quads
(unfortunately that comes from a java step - you have to do the
calculation).
It isn't as chatty as tdbloader because it is using external tools.
Andy
---
A. Soroka
The University of Virginia Library
[1] https://seaborne.blogspot.com/2010/12/repacking-btrees.html
On Oct 28, 2016, at 11:12 AM, Andy Seaborne <[email protected]> wrote:
On 28/10/16 15:58, A. Soroka wrote:
That's right-- if I understand it correctly, there are two steps--
POSIX sort to develop the index orderings, and then packing the
actual index files.
For the POSIX sort step, it's certainly true that more parallelism
than needed would be a bad thing. With Andy's help I just made a
commit that allows a little more control using the common --parallel
flag for sort. But the current ergonomics seem suboptimal. E.g. with
the current settings indexing a 300Mt dataset on a 24-core box with
fast storage, I saw only one core in full use, and very little IO
usage. Aliasing in some parallelism via the sort flag brought several
more cores into play and cut the time spent by two-thirds.
What rates are you getting?
I don't
know how normal that is, but for the sort step, my argument is not
that we could find universally better ergonomics, but that we could
bake some flexibility in for those who want to try adjustments on
their particular hardware, including the ability to try running
multiple sorts at one time.
That sounds like an interesting experiment to carry out and if successful
change the released code.
For the other step, I don't feel like I understand the index-packing
code well enough yet to form an opinion, which is one reason for the
question. It seems that it could run in parallel without difficulty,
but maybe I don't understand the relationships between the indexes
well enough.
Index packing is I/O bound and is sequential. There is little computation going
on.
Doing two packings in parallel would break up the sequential write sequence so
there would be need to be a noticable gain in some way to compensate for the
impact.
Bus contention when it's a SSD may come into play. The quality/speed of the
connection to the SSD is related to how much $$$ the server cost!
Another question then would be: maybe we could split the current
'index' phase into 'order' and 'pack' phases, again for those who
would like to try tuning each step for their situation?
Interesting possibility - needs trying out and bedding down before it goes into
the standard release scripts IMO. What works well in one environment may not
in another. Lots of options suits some people and not others.
Andy
---
A. Soroka
The University of Virginia Library
On Oct 28, 2016, at 10:24 AM, Rob Vesse <[email protected]> wrote:
If memory serves those are the phases that use POSIX sort right?
Sort will try and do an in-memory sort as far as possible and fall back to a
disk-based merge sort if not. Also we usually configure sort to run in parallel
If you try to process different indexing in parallel you would create a lot of
memory and disk contention which would likely slowdown overall performance
For sufficiently large data sets there is also a risk of exhausting disk space
during the sort phase and building multiple indexes in parallel would only
exacerbate this
Rob
On 28/10/2016 14:33, "A. Soroka" <[email protected]> wrote:
I'm still learning about tdbloader2 and have another question about the index
phase: is there any reason why the processes for the various index orderings
(SPO, GSPO, etc.) couldn't go on in parallel? Or am I missing some switch or
setting that already allows that?
---
A. Soroka
The University of Virginia Library
