Thank you Ulrich!

Ulrich Windl schrieb:
> Well if you thing the problem is transport related, and you are doing 
> performance 
> evaluation anyway, you could try to use iozone: I think it has a 
> checksumming/verification feature. Maybe this helps to reveal the problems. 
> Maybe 
> even, write your own code (e.g. write some long piece of data to a partition 
> directly (using no file system), then read it back and compare it 
> byte-by-byte.
>   
We digged deeper into the material and we found the following stunning results:

* The ocurrance of the error depends on the amount of avaible RAM on the 
initiator Machine.
* We regulary boot our Machines limited to 256MB RAM for IO-Testing. 
* But if we give the machine 4 GB RAM the error vanishes.

But the error itself depends not on the amount of moved data. 
With 256MB RAM on the initiator "tiotest -f 1200 -t4" wich uses 4.8GB-IO-Space 
fails.
With 4 GB of RAM even with a iSCSI-Devive of 100GB and TioTest on 50GB not a 
single error occures. 

Is there anybody out there with IB to reproduce this error?
This may be of interest for the infiniband community as well as the 
iSCSI-Community.

Our Test-System consist out of two servers

* supermicro H8DME-2 board
* 2 x AMD Opteron Quadcore 2356
* 32 GB RAM
* HCA : Mellanox Technologies MT25204 [InfiniHost III Lx HCA] (rev 2) tested 
back-to-back as well as over flextronic IB-switch.

Identically BIOS, BIOS-Verson, OS setup on target and intitiator side.

On the target-Side we use the actual stgtd 0.9.3 compiled against the debian 
lenny IB-dev-libs. 

Target (ATHENE):
tgtadm --lld iscsi --op new --mode target --tid 1 -T de.inqbus.athene:test
tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /dev/vg0/test
tgtadm --lld iscsi --op bind --mode target --tid 1 -I 10.1.3.0/24


Initiator (ARES):

ares:~# iscsi_discovery 10.1.3.32 -tiser
iscsiadm: No active sessions.
Set target de.inqbus.athene:test to automatic login over iser to portal 
10.1.3.32:3260
Logging out of session [sid: 3, target: de.inqbus.athene:test, portal: 
10.1.3.32,3260]
Logout of [sid: 3, target: de.inqbus.athene:test, portal: 10.1.3.32,3260]: 
successful
discovered 1 targets at 10.1.3.32
ares:~# iscsi_discovery 10.1.3.32 -tiser -l
iscsiadm: No active sessions.
Set target de.inqbus.athene:test to automatic login over iser to portal 
10.1.3.32:3260
discovered 1 targets at 10.1.3.32
ares:~# mkfs.ext3 
/dev/disk/by-path/ip-10.1.3.32:3260-iscsi-de.inqbus.athene:test-lun-1
ares:~# mount 
/dev/disk/by-path/ip-10.1.3.32:3260-iscsi-de.inqbus.athene:test-lun-1 /mnt/test
ares:~# cd /mnt/test
ares:/mnt/test# tiotest -f1200 -t4
Tiotest results for 4 concurrent io threads:
,----------------------------------------------------------------------.
| Item                  | Time     | Rate         | Usr CPU  | Sys CPU |
+-----------------------+----------+--------------+----------+---------+
| Write        4800 MBs |   22.2 s | 215.801 MB/s |  -5.1 %  | 423.4 % |
| Random Write   16 MBs |    0.1 s | 163.298 MB/s |   8.4 %  | 255.0 % |
| Read         4800 MBs |    9.7 s | 492.869 MB/s |   6.1 %  | 134.5 % |
| Random Read    16 MBs |    0.1 s | 191.520 MB/s | -29.4 %  | 181.4 % |
`----------------------------------------------------------------------'
Tiotest latency results:
,-------------------------------------------------------------------------.
| Item         | Average latency | Maximum latency | % >2 sec | % >10 sec |
+--------------+-----------------+-----------------+----------+-----------+
| Write        |        0.059 ms |     2285.150 ms |  0.00008 |   0.00000 |
| Random Write |        0.008 ms |        0.254 ms |  0.00000 |   0.00000 |
| Read         |        0.030 ms |      709.000 ms |  0.00000 |   0.00000 |
| Random Read  |        0.076 ms |        2.059 ms |  0.00000 |   0.00000 |
|--------------+-----------------+-----------------+----------+-----------|
| Total        |        0.045 ms |     2285.150 ms |  0.00004 |   0.00000 |
`--------------+-----------------+-----------------+----------+-----------'

ares:~# demsg
[ 1069.687429] EXT3-fs error (device sdc): ext3_free_blocks_sb: bit already 
cleared for block 176656
[ 1069.698140] EXT3-fs error (device sdc): ext3_free_blocks_sb: bit 
..
 
We use the actual rc of Debian lenny. Open-iscsi, kernel and kernel-Modules 
from Debian. User-Space-IB-Tools are homebrewed from OFED 1.3. The error 
reproduces with OFED 1.4.

Last Question: Where to file this error best?

Any help welcome.

Best Regards

Volker


-- 
====================================================
   inqbus it-consulting      +49 ( 341 )  5643800
   Dr.  Volker Jaenisch      http://www.inqbus.de
   Herloßsohnstr.    12      0 4 1 5 5    Leipzig
   N  O  T -  F Ä L L E      +49 ( 170 )  3113748
====================================================


--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google Groups 
"open-iscsi" group.
To post to this group, send email to open-iscsi@googlegroups.com
To unsubscribe from this group, send email to 
open-iscsi+unsubscr...@googlegroups.com
For more options, visit this group at http://groups.google.com/group/open-iscsi
-~----------~----~----~----~------~----~------~--~---

Reply via email to