[HACKERS] [GSOC 17] Eliminate O(N^2) scaling from rw-conflict tracking in serializable transactions
Hi, Alvaro and Kevin. I'm Mengxing. This is a “synchronization” email to tell you what I've done and my next plan. I'm looking forward to your advice. According to my proposal, I want to prepare the experimental environment during the community bonding period. As this is the first time I discuss with Alvaro, here I will introduce the environment again. My lab have a Lenovo System x3950 X6 machine. https://www.lenovo.com/images/products/system-x/pdfs/datasheets/x3950_x6_ds.pdf More specifically, there are 8 sockets, each has 15 Intel(R) Xeon(R) CPU E7-8870 v2 @ 2.30GHz. Thus we have 120 cores in total. The storage is a 1 TB SSD, with SAS interface, 500MB write bandwidth. OS is ubuntu 14.04. I've compiled & installed PostgreSQL and have tested it by using pgbench. I tuned parameter for database according to http://www.dbdude.com/postgresql/postgresqlperformance.php#BESTPRACTICES For pgbench, I set scale factor (-s) 512. Other configurations are default values. Because we have too many cores, SSD becomes the bottleneck. In my test, pgbench can scale to 36 connections. ( 18 KTPS throughput). CPU utilization is smaller than 30%. Therefore: 1. Is there anything wrong in my tuning parameters？For example, should I close "fsync"? Because we don't care recovery here. 2. Can we use just two sockets (30 physical cores) to run database server? Then the CPU can be the bottleneck so that it shows the problem we try to solve. BTW. Here is the question of Stephen. > What method of communication will be used among the mentors and with > Mengxing. What method do you prefer? > Frequency of status updates (must be at least once a week and more often is > encouraged). How about reporting my status once a week? > What steps will be taken next during the community bonding period. As I wrote in the proposal, I want to establish the environment and read the related source code. Do you have any suggestions for me? -- Mengxing Liu
[HACKERS] [GSOC 17] Eliminate O(N^2) scaling from rw-conflict tracking in serializable transactions
Hi, all My name is Mengxing Liu. I am interested in the project "Eliminate O(N^2) scaling from rw-conflict tracking in serializable transactions”. After discussing with Kevin off-list, I think it's time to post discussion here. I am afraid that there are two things that I need your help. Thank you very much. > > > So the task is clear. We can use a tree-like or hash-like data > > structure to speed up this function. > > Right -- especially with a large number of connections holding a > large number of conflicts. In one paper with high concurrency, they > found over 50% of the CPU time for PostgreSQL was going to these > functions (including functions called by them). This seems to me to > be due to the O(N^2) (or possibly worse) performance from the number > of connections. Anyone knows the title of this paper? I want to reproduce its workloads. > > Remember, I think most of the work here is going to be in > benchmarking. We not only need to show improvements in simple test > cases using readily available tools like pgbench, but think about > what types of cases might be worst for the approach taken and show > that it still does well -- or at least not horribly. It can be OK > to have some slight regression in an unusual case if the common > cases improve a lot, but any large regression needs to be addressed > before the patch can be accepted. There are some community members > who are truly diabolical in their ability to devise "worst case" > tests, and we don't want to be blind-sided by a bad result from one > of them late in the process. > Are there any documents or links introducing how to test and benchmark PostgreSQL? I may need some time to learn about it. Thanks. -- Mengxing Liu -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
