Re: [PATCH v2] Support for write stream IDs

2015-05-11 Thread Martin K. Petersen
> "Jens" == Jens Axboe writes: Jens, Jens> There are actual technical challenges on the device side that Jens> sometimes interferes. [...] Right now we use the same protocol to speak to USB keys and million dollar storage arrays. That's because the protocol was designed to be abstract and

Re: [PATCH v2] Support for write stream IDs

2015-05-11 Thread Martin K. Petersen
Jens == Jens Axboe ax...@fb.com writes: Jens, Jens There are actual technical challenges on the device side that Jens sometimes interferes. [...] Right now we use the same protocol to speak to USB keys and million dollar storage arrays. That's because the protocol was designed to be abstract

Re: [PATCH v2] Support for write stream IDs

2015-05-08 Thread Jens Axboe
On 05/07/2015 01:19 PM, Martin K. Petersen wrote: "Jens" == Jens Axboe writes: Jens> This wont solve the problem of devices having too few streams. But Jens> it'll work regardless, we'll just have to push them separately to Jens> do that. It's not an easy problem for them either, resource

Re: [PATCH v2] Support for write stream IDs

2015-05-08 Thread Jens Axboe
On 05/07/2015 01:19 PM, Martin K. Petersen wrote: Jens == Jens Axboe ax...@fb.com writes: Jens This wont solve the problem of devices having too few streams. But Jens it'll work regardless, we'll just have to push them separately to Jens do that. It's not an easy problem for them either,

Re: [PATCH v2] Support for write stream IDs

2015-05-07 Thread Martin K. Petersen
> "Jens" == Jens Axboe writes: Jens> This wont solve the problem of devices having too few streams. But Jens> it'll work regardless, we'll just have to push them separately to Jens> do that. It's not an easy problem for them either, resource Jens> constraints on the device side could exclude

Re: [PATCH v2] Support for write stream IDs

2015-05-07 Thread Martin K. Petersen
Jens == Jens Axboe ax...@fb.com writes: Jens This wont solve the problem of devices having too few streams. But Jens it'll work regardless, we'll just have to push them separately to Jens do that. It's not an easy problem for them either, resource Jens constraints on the device side could

Re: [PATCH v2] Support for write stream IDs

2015-05-06 Thread Jens Axboe
On 05/06/2015 08:26 AM, Peter Zijlstra wrote: On Tue, May 05, 2015 at 06:09:18PM -0400, Martin K. Petersen wrote: "Jens" == Jens Axboe writes: Jens> I'm not trying to make a shortcut. I deliberately do not want to Jens> make ID generation/assignment part of the kernel. There's no Jens>

Re: [PATCH v2] Support for write stream IDs

2015-05-06 Thread Jens Axboe
On 05/05/2015 04:09 PM, Martin K. Petersen wrote: "Jens" == Jens Axboe writes: Jens> I'm not trying to make a shortcut. I deliberately do not want to Jens> make ID generation/assignment part of the kernel. There's no Jens> reason that can't exist outside of the kernel, in a libstreamid or

Re: [PATCH v2] Support for write stream IDs

2015-05-06 Thread Boaz Harrosh
On 05/06/2015 01:09 AM, Martin K. Petersen wrote: >> "Jens" == Jens Axboe writes: <> > > The only sensible solution is for the kernel to manage the stream > IDs. And for them to be plentiful. The storage device is free to ignore > them, do LRU or whatever it pleases to manage them if it has

Re: [PATCH v2] Support for write stream IDs

2015-05-06 Thread Peter Zijlstra
On Tue, May 05, 2015 at 06:09:18PM -0400, Martin K. Petersen wrote: > > "Jens" == Jens Axboe writes: > > Jens> I'm not trying to make a shortcut. I deliberately do not want to > Jens> make ID generation/assignment part of the kernel. There's no > Jens> reason that can't exist outside of the

Re: [PATCH v2] Support for write stream IDs

2015-05-06 Thread Peter Zijlstra
On Tue, May 05, 2015 at 06:09:18PM -0400, Martin K. Petersen wrote: Jens == Jens Axboe ax...@fb.com writes: Jens I'm not trying to make a shortcut. I deliberately do not want to Jens make ID generation/assignment part of the kernel. There's no Jens reason that can't exist outside of the

Re: [PATCH v2] Support for write stream IDs

2015-05-06 Thread Boaz Harrosh
On 05/06/2015 01:09 AM, Martin K. Petersen wrote: Jens == Jens Axboe ax...@fb.com writes: The only sensible solution is for the kernel to manage the stream IDs. And for them to be plentiful. The storage device is free to ignore them, do LRU or whatever it pleases to manage them if it has an

Re: [PATCH v2] Support for write stream IDs

2015-05-06 Thread Jens Axboe
On 05/06/2015 08:26 AM, Peter Zijlstra wrote: On Tue, May 05, 2015 at 06:09:18PM -0400, Martin K. Petersen wrote: Jens == Jens Axboe ax...@fb.com writes: Jens I'm not trying to make a shortcut. I deliberately do not want to Jens make ID generation/assignment part of the kernel. There's no

Re: [PATCH v2] Support for write stream IDs

2015-05-06 Thread Jens Axboe
On 05/05/2015 04:09 PM, Martin K. Petersen wrote: Jens == Jens Axboe ax...@fb.com writes: Jens I'm not trying to make a shortcut. I deliberately do not want to Jens make ID generation/assignment part of the kernel. There's no Jens reason that can't exist outside of the kernel, in a libstreamid

Re: [PATCH v2] Support for write stream IDs

2015-05-05 Thread Martin K. Petersen
> "Jens" == Jens Axboe writes: Jens> I'm not trying to make a shortcut. I deliberately do not want to Jens> make ID generation/assignment part of the kernel. There's no Jens> reason that can't exist outside of the kernel, in a libstreamid or Jens> similar. That just perpetuates the broken

Re: [PATCH v2] Support for write stream IDs

2015-05-05 Thread Jens Axboe
On 05/05/2015 03:39 PM, Martin K. Petersen wrote: "Jens" == Jens Axboe writes: Jens> The kernel patches deal only with ensuring that the stream Jens> information gets passed down. If the device requires explicit Jens> stream open/close actions, then that needs to be handled on the Jens> side.

Re: [PATCH v2] Support for write stream IDs

2015-05-05 Thread Martin K. Petersen
> "Jens" == Jens Axboe writes: Jens> The kernel patches deal only with ensuring that the stream Jens> information gets passed down. If the device requires explicit Jens> stream open/close actions, then that needs to be handled on the Jens> side. Well, I'm deeply concerned about that "on the

Re: [PATCH v2] Support for write stream IDs

2015-05-05 Thread Jens Axboe
On 05/05/2015 02:51 PM, Jeff Moyer wrote: Jens Axboe writes: Hi, Changes since the last posting: - Added a specific per-file fadvise setting. POSIX_FADV_STREAMID sets the inode and file stream ID, POSIX_FADV_STREAMID_FILE sets just the file stream ID. - Addressed review comments.

Re: [PATCH v2] Support for write stream IDs

2015-05-05 Thread Jeff Moyer
Jens Axboe writes: > Hi, > > Changes since the last posting: > > - Added a specific per-file fadvise setting. POSIX_FADV_STREAMID sets > the inode and file stream ID, POSIX_FADV_STREAMID_FILE sets just the > file stream ID. > > - Addressed review comments. > > I've since run some testing

Re: [PATCH v2] Support for write stream IDs

2015-05-05 Thread Christoph Hellwig
On Tue, May 05, 2015 at 02:31:34PM -0600, Jens Axboe wrote: > There is a user, we are using it. And there's no data structure bloating, > both the file and inode additions are filling existing holes. FYI, they do increase the size on any 32-bit architecture, which is where it hurts most. -- To

Re: [PATCH v2] Support for write stream IDs

2015-05-05 Thread Christoph Hellwig
On Tue, May 05, 2015 at 02:31:34PM -0600, Jens Axboe wrote: > Even outside of that, there are use cases for caching that need not have > hardware assist. Could, would. But in the meantime you'd adding dead wood kernel code, and even worse user interfaces. > >Merging infrastructure without any

Re: [PATCH v2] Support for write stream IDs

2015-05-05 Thread Jens Axboe
On 05/05/2015 02:20 PM, Christoph Hellwig wrote: On Tue, May 05, 2015 at 02:12:01PM -0600, Jens Axboe wrote: We can't merge the NVMe bits until the proposal is included/finalized. But this is a problem. I don't want to add this to the Facebook kernel until we know the API is stable, while I

Re: [PATCH v2] Support for write stream IDs

2015-05-05 Thread Christoph Hellwig
On Tue, May 05, 2015 at 02:12:01PM -0600, Jens Axboe wrote: > We can't merge the NVMe bits until the proposal is included/finalized. But > this is a problem. I don't want to add this to the Facebook kernel until we > know the API is stable, while I have no problem adding experimental NVMe >

Re: [PATCH v2] Support for write stream IDs

2015-05-05 Thread Jens Axboe
On 05/05/2015 02:07 PM, Christoph Hellwig wrote: On Tue, May 05, 2015 at 02:02:54PM -0600, Jens Axboe wrote: Unless there are any grave concerns here, I'd like to merge this for 4.2. Without a driver implementing the feature I don't see any reason to even consider merging it. We can't merge

Re: [PATCH v2] Support for write stream IDs

2015-05-05 Thread Christoph Hellwig
On Tue, May 05, 2015 at 02:02:54PM -0600, Jens Axboe wrote: > Unless there are any grave concerns here, I'd like to merge this for > 4.2. Without a driver implementing the feature I don't see any reason to even consider merging it. -- To unsubscribe from this list: send the line "unsubscribe

[PATCH v2] Support for write stream IDs

2015-05-05 Thread Jens Axboe
Hi, Changes since the last posting: - Added a specific per-file fadvise setting. POSIX_FADV_STREAMID sets the inode and file stream ID, POSIX_FADV_STREAMID_FILE sets just the file stream ID. - Addressed review comments. I've since run some testing with write streams. Test case was a

Re: [PATCH v2] Support for write stream IDs

2015-05-05 Thread Martin K. Petersen
Jens == Jens Axboe ax...@fb.com writes: Jens I'm not trying to make a shortcut. I deliberately do not want to Jens make ID generation/assignment part of the kernel. There's no Jens reason that can't exist outside of the kernel, in a libstreamid or Jens similar. That just perpetuates the broken

Re: [PATCH v2] Support for write stream IDs

2015-05-05 Thread Christoph Hellwig
On Tue, May 05, 2015 at 02:12:01PM -0600, Jens Axboe wrote: We can't merge the NVMe bits until the proposal is included/finalized. But this is a problem. I don't want to add this to the Facebook kernel until we know the API is stable, while I have no problem adding experimental NVMe changes

[PATCH v2] Support for write stream IDs

2015-05-05 Thread Jens Axboe
Hi, Changes since the last posting: - Added a specific per-file fadvise setting. POSIX_FADV_STREAMID sets the inode and file stream ID, POSIX_FADV_STREAMID_FILE sets just the file stream ID. - Addressed review comments. I've since run some testing with write streams. Test case was a

Re: [PATCH v2] Support for write stream IDs

2015-05-05 Thread Jens Axboe
On 05/05/2015 02:20 PM, Christoph Hellwig wrote: On Tue, May 05, 2015 at 02:12:01PM -0600, Jens Axboe wrote: We can't merge the NVMe bits until the proposal is included/finalized. But this is a problem. I don't want to add this to the Facebook kernel until we know the API is stable, while I

Re: [PATCH v2] Support for write stream IDs

2015-05-05 Thread Christoph Hellwig
On Tue, May 05, 2015 at 02:31:34PM -0600, Jens Axboe wrote: Even outside of that, there are use cases for caching that need not have hardware assist. Could, would. But in the meantime you'd adding dead wood kernel code, and even worse user interfaces. Merging infrastructure without any users

Re: [PATCH v2] Support for write stream IDs

2015-05-05 Thread Christoph Hellwig
On Tue, May 05, 2015 at 02:02:54PM -0600, Jens Axboe wrote: Unless there are any grave concerns here, I'd like to merge this for 4.2. Without a driver implementing the feature I don't see any reason to even consider merging it. -- To unsubscribe from this list: send the line unsubscribe

Re: [PATCH v2] Support for write stream IDs

2015-05-05 Thread Jens Axboe
On 05/05/2015 02:07 PM, Christoph Hellwig wrote: On Tue, May 05, 2015 at 02:02:54PM -0600, Jens Axboe wrote: Unless there are any grave concerns here, I'd like to merge this for 4.2. Without a driver implementing the feature I don't see any reason to even consider merging it. We can't merge

Re: [PATCH v2] Support for write stream IDs

2015-05-05 Thread Jeff Moyer
Jens Axboe ax...@fb.com writes: Hi, Changes since the last posting: - Added a specific per-file fadvise setting. POSIX_FADV_STREAMID sets the inode and file stream ID, POSIX_FADV_STREAMID_FILE sets just the file stream ID. - Addressed review comments. I've since run some testing

Re: [PATCH v2] Support for write stream IDs

2015-05-05 Thread Christoph Hellwig
On Tue, May 05, 2015 at 02:31:34PM -0600, Jens Axboe wrote: There is a user, we are using it. And there's no data structure bloating, both the file and inode additions are filling existing holes. FYI, they do increase the size on any 32-bit architecture, which is where it hurts most. -- To

Re: [PATCH v2] Support for write stream IDs

2015-05-05 Thread Jens Axboe
On 05/05/2015 03:39 PM, Martin K. Petersen wrote: Jens == Jens Axboe ax...@fb.com writes: Jens The kernel patches deal only with ensuring that the stream Jens information gets passed down. If the device requires explicit Jens stream open/close actions, then that needs to be handled on the Jens

Re: [PATCH v2] Support for write stream IDs

2015-05-05 Thread Jens Axboe
On 05/05/2015 02:51 PM, Jeff Moyer wrote: Jens Axboe ax...@fb.com writes: Hi, Changes since the last posting: - Added a specific per-file fadvise setting. POSIX_FADV_STREAMID sets the inode and file stream ID, POSIX_FADV_STREAMID_FILE sets just the file stream ID. - Addressed review

Re: [PATCH v2] Support for write stream IDs

2015-05-05 Thread Martin K. Petersen
Jens == Jens Axboe ax...@fb.com writes: Jens The kernel patches deal only with ensuring that the stream Jens information gets passed down. If the device requires explicit Jens stream open/close actions, then that needs to be handled on the Jens side. Well, I'm deeply concerned about that on the

[PATCH v2] Support for write stream IDs

2015-04-18 Thread Jens Axboe
Hi, v2 of this posting. Changes since v1: - Rebased on top of current master. - Fix EINVAL -> -EINVAL typo. - Cleanup up BIO_STREAM_OFFSET definition. - Pack i_streamid and f_streamid better into struct file and struct inode. - Add a separate per-file hint, FADV_FILE_STREAMID. This only

[PATCH v2] Support for write stream IDs

2015-04-18 Thread Jens Axboe
Hi, v2 of this posting. Changes since v1: - Rebased on top of current master. - Fix EINVAL - -EINVAL typo. - Cleanup up BIO_STREAM_OFFSET definition. - Pack i_streamid and f_streamid better into struct file and struct inode. - Add a separate per-file hint, FADV_FILE_STREAMID. This only