On 22/2/2024 06:42, Thomas Munro wrote:
On Wed, Feb 21, 2024 at 7:34 PM Andrei Lepikhov
<a.lepik...@postgrespro.ru> wrote:
I see in [1] that the reporter mentioned a delay between the error
message in parallel HashJoin and the return control back from PSQL. Your
patch might reduce this delay.
Also, I have the same complaint from users who processed gigabytes of
data in parallel HashJoin. Presumably, they also stuck into the unlink
of tons of temporary files. So, are you going to do something with this
code?
Yeah, right. I will aim to get this into the tree next week. First,
there are a couple of minor issues to resolve around freeing that
Heikki mentioned. Then there is the question of whether we think this
might be a candidate for back-patching, given the complaints you
mention. Opinions?
The code is related to performance, not a bug. Also, it adds one
external function into the 'sharedtuplestore.h'. IMO, it isn't worth it
to make back-patches.
I would add that the problems you reach when you get to very large
number of partitions are hard (see several very long threads about
extreme skew for one version of the problem, but even with zero/normal
skewness and perfect estimation of the number of partitions, if you
ask a computer to partition 42TB of data into partitions that fit in a
work_mem suitable for a Commodore 64, it's gonna hurt on several
levels) and this would only slightly improve one symptom. One idea
that might improve just the directory entry and file descriptor
aspect, would be to scatter the partitions into (say) 1MB chunks
within the file, and hope that the file system supports holes (a bit
like logtape.c's multiplexing but I wouldn't do it quite like that).
Thanks, I found in [1] good entry point to dive into this issue.
[1]
https://www.postgresql.org/message-id/CA+hUKGKDbv+5uiJZDdB1wttkMPFs9CDb6=02qxitq4am-kb...@mail.gmail.com
--
regards,
Andrei Lepikhov
Postgres Professional
