Looks like you're running out of adapter resources for storing the
physical buffers lists (PBLs).
The cards have 32MB of memory for storing PBLs.
32MB of memory can store 4MB of 8B physical addresses.
For user mode, your page size is 4KB, so that maps out to 4MB * 4KB =
16GB of memory.
Hey Felix,
Can we somehow adjust the PMRX memory map dynamically? For instance,
maybe drop the number of TPTs to allow for more PBL memory?
Steve
pandit ib wrote:
Hi,
In our testing we are hitting a memory registration limit of 16GB with
the Chelsio cards.
Here are some sample results for different region sizes.
region size Number of Successful registration Total
2MB 8178
~16GB
16MB 1021
~16GB
128MB 127
~16GB
Are there any knobs that we can use to raise that limit?
For example, on mlx4 we use "log_num_mtt" module load parameter to
increase the limit.
Here is a sample output of the test:
[r...@lv2 ~]# ./mr-test 16777216
mr-test: bufsize 16777216
device # 0 name="cxgb3_0" guid="00074305ca890000"
ibv_open_device() context=0x77b5100
ibv_alloc_pd() pd=0x77b50e0
alloc: 1021
ibv_reg_mr failed:: Cannot allocate memory
fw_ver: 7.4.0
max_mr_size 0x100000000
max_mr: 32768, could only register 1021 regions; Region size 16MB
sleep 5 sec
free: 0
done
_______________________________________________
general mailing list
general@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
_______________________________________________
general mailing list
general@lists.openfabrics.org
http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
