Boaz Harrosh wrote:
> Hi Mike, list.
> Mike Christie has pointed out of a serious problem for us which we need
> the list help of.
> It started with a question by Ulrich Windl of why data-digests are
> not supported/recommended by open-iscsi installations and distros.
> [iscsi data-digests is when the complete payload of an iscsi transaction
>  initiator-target is signed by an HMAC(SHA1) both read/write]
> Mike Christie wrote:
>> Ulrich Windl wrote:
>> Data digests were working but when upstream did the scatterlist changes 
>> to the kernel it broke data digests. We have not found the cause yet.
>> For Red Hat, they do not support them for different reasons depending on 
>> the version and arch. For example in RHEL5, the big endien crypto digest 
>> code is busted. It needs a fix from upstream, and I think in general 
>> there is still some other bugs in the digest code.
>>> I see the performance impact, but is there another reason against 
>>> implementing it? 
>>> Can I safely activate it on the target, or will it cause problems?
>> Another reason a lot of distros do not support it is because a common 
>> problem we always hit is that users will write out some data, then start 
>> modifying it again. But the kernel will normally not do do a sync write 
>> when you do a write. So once the write() returns, the kernel is still 
>> sending it through the caches, block, scsi, and iscsi layers. If you are 
>> writing to the data while the it is working its way through the iscsi 
>> layers, the iscsi layer could have done the digest calculation, then you 
>> could modify it and now when the target checks it the digest check will 
>> fail. And so this happens over and over and you get digest errors all 
>> over the place and the iscsi layers fire their error handling and retry 
>> and retry, and in the end they just say forget it and do not support 
>> data digests.
> Mike if what you said in the last paragraph is true, about FS modifying the 
> data
> while the request is in-flight, then it does not explain your statement above
> about, things getting worse around the scatterlist changes.

They are two separate issues.

Around the time of the scatterlist changes I will get an oops in the 
digest calculation code (when we call into the crypto callouts), or in 
newer kernels the oops went away and now I will get data digest errors. 
I am still trying to narrow down the commit and line and make sure that 
the oops is fixed and did not turn into a digest error or if maybe I am 
hitting a real digest error.

The second issue is that we normally do zero copy for writes. I do not 
think it is FS bug or net bug or a bug in the iscsi layer. Maybe more of 
a bug in what the user expects (who reads the man page for write() to 
check if the data is committed to disk when write() returns). We 
discussed this a couple times. For open-iscsi we tried to close the gap, 
by not doing zero copy writes when data digests are used. And a long 
long time ago this was discussed for linux-iscsi, and I think that is 
one of the reasons we added DID_IMM_RETRY to the scsi layer (we can then 
avoid the 5 retry limit in this case and retry until it is resolved).

> The way I see it there can be two fundamental problems:
> 1. The FS is permitted to (or sinfully) modifies pages of memory while a 
> request to
>    write these pages is already in-flight. fsdevel guys might want to comment 
> on that?
>    Mike have you observed these problems with a particular file system?
>    I can anticipate such a problem arising in a memory-mapped IO, while a 
> page-cache
>    write-back is in progress. Is that so? is Linux not safe in this regard? 
> If so
>    how does DM & MD do there raid parity calculations? do they copy the data?
> 2. iSCSI releases the request too soon, before the all data was actually used 
> up by the
>    network stack, and is allowing the FS to continue modifying these pages.
>    This is a serious problem which means that there can be crashes and data 
> corruption even
>    if data-digest are not used.
>    Actually we did move not long ago from copy of network data to been 
> completely copy-less
>    could that be the point in time things stopped working?
> 3. Plain coding bug, but I could not find any.
> I know in the passed that data-digests are a grate tool for finding bugs that 
> otherwise can
> go undetected, it happened to me several times in the passed. All of these 
> cases reviled a flaw
> in the code, do to rebasing, things changing, plain programmer bugs.
> Mike, I'm running here a plain iscsi initiator-target setup and the 
> regression tests, and it
> runs. What setup and tests did you run to trigger these digest retries, I 
> would like to
> reproduce this here, and investigate.

The open-iscsi/test regression script and dat file. Once the section 
with data digests runs I hit the oops/digest error. I am not sure if I 
ever hit the second zero copy write issues. I might be hitting that now. 
Like I said, I have not had time to check if the oops turned into a 
digest error when it should not or if I am hitting the zero copy issue.

You received this message because you are subscribed to the Google Groups 
"open-iscsi" group.
To post to this group, send email to
To unsubscribe from this group, send email to
For more options, visit this group at

Reply via email to