Re: The future of ZFS in FreeBSD

[Matthew Macy]

On Thu, Dec 20, 2018 at 03:11 Eugene M. Zheganin  wrote:

> Hello,
>
> On 19.12.2018 23:32, Allan Jude wrote:
> > The biggest thing to remember is that this is still OpenZFS, and still
> > run by the same developers as it has been. We are just commonizing on
> > the repo that has the most features integrated into it.
>
> Does it mean that ZoF and thus FreeBSD will lose NFSv4 ACLs because
> there is no such thing in ZoL ?



No, but the ZTS will need to have ACL tests re-added. Please help out by
aiding in getting this merged:

https://github.com/zfsonlinux/zfs/pull/7728

Thanks.
-M
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: The future of ZFS in FreeBSD

[Matthew Macy]

On Thu, Dec 20, 2018 at 06:39 Alexey Dokuchaev  wrote:

> On Thu, Dec 20, 2018 at 03:49:38PM +0500, Eugene M. Zheganin wrote:
> > On 19.12.2018 23:32, Allan Jude wrote:
> > > The biggest thing to remember is that this is still OpenZFS, and still
> > > run by the same developers as it has been. We are just commonizing on
> > > the repo that has the most features integrated into it.
> >
> > Does it mean that ZoF and thus FreeBSD will lose NFSv4 ACLs because
> > there is no such thing in ZoL?


> +1.  I'm also worried if this would bring more Linuxish bits into our
> kernel (cf. LinuxKPI).  Also, I thought that ZFS was never really native
> to Linux but implemented through SPL (Solaris Porting Layer), and Linux'
> VFS is not ARC-aware unlike Solaris and FreeBSD.
>

There is no LinuxKPI involved here. Please re-read Allan’s comments on
directory structuring. No open source OS supporting ZFS has a VM subsystem
that is integrated with the ARC. The limited feedback that the ARC has from
FreeBSD’s VM will remain unchanged.



> It would be quite upsetting to see ZFS as we know it in FreeBSD become
> pessimized because of those things. :-(
>

ZFS will be no more pessimized than it currently is. Talk to mjg some time
about ZFS some time, there are ... scaling issues in its locking
strategies. Apart from that FreeBSD’s VFS itself has serious scaling. None
of this will get better or worse when we change the vendor repo we use for
integrating changes.

-M
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: The future of ZFS in FreeBSD

[Warner Losh]

On Thu, Dec 20, 2018 at 1:49 PM Allan Jude  wrote:

> I am to give a big thanks to Matt Ahrens for organizing the monthly
> OpenZFS Leadership meeting, and the OpenZFS developer summit, and to
> Brian Behlendorf for being so helpful, and willing to work to make
> OpenZFS better for everyone.
>

I'd recommend the monthly leadership meetings. They dispelled any doubt
that I had that there were overly powerful factions present that would
disadvantage FreeBSD. The calls were worthwhile to understand the context
of a number of other things being discussed. It seems much healthier in
this context than things I've seen in the Illumos context because of a lack
of burdensome process that provided no benefit and actually got in the way
(which honestly, was my expectations going into the meeting).

There's good reason to trust OpenZFS, it's leadership and decision making
process. If ZoL is the best upstream source, then I trust their judgement
on that, regardless of the hassles it will cause us in the coming months.
>From the discussions I've seen, there is no pain-free choice that doesn't
relegate FreeBSD to a backwater of what we have today.

Warner
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: The future of ZFS in FreeBSD

[Allan Jude]

On 12/19/2018 13:32, Allan Jude wrote:

Today, the OpenZFS repo is just a fork of the illumos-gate repo, but
where pull requests are accepted, and where previous Delphix employees


This should say 'previously', Prakash Surya and Matt Ahrens still work 
at Delphix.



would deal with the process of trying to upstream patches to illumos.
This process has not worked well recently, as things have gotten stuck
waiting for 'merge advocates' in illumos.



The major stumbling block was the lack of modern test infrastructure.
Obviously depending on one or two people to merge stuff was a bottleneck 
as those people inevitably get busy.




Anyway, the point is that this doesn't make FreeBSD any more dependent 
on Linux. The goal is this project is to continue to make it easier and 
easier to port OpenZFS and its new features to all platforms (illumos, 
FreeBSD, Linux, OS X, Windows, NetBSD, etc), and to ensure that new 
features arrive in a timely fashion in all of the platforms.


Unifying on a single CI setup saves duplication of effort, and ensures 
that changes can easily be tested on all of the platforms.


This is a project is a good thing, it ensures that FreeBSD will keep 
current on OpenZFS going forward, and helps improve the entire OpenZFS 
ecosystem.


I am to give a big thanks to Matt Ahrens for organizing the monthly 
OpenZFS Leadership meeting, and the OpenZFS developer summit, and to 
Brian Behlendorf for being so helpful, and willing to work to make 
OpenZFS better for everyone.


--
Allan Jude
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: The future of ZFS in FreeBSD

[Matthew Macy]

On Thu, Dec 20, 2018 at 9:33 AM Warner Losh  wrote:
>
>
> Matt,
>
> This is a fairly comprehensive plan. Kudos for putting it together.
>
> The big question here is do you have a complete list of FreeBSD-specific 
> changes that will be lost in the cut-over? We've heard about TRIM support and 
> maybe NFSv4, but are there others that can be identified?
>
> Once you have that list, it wouldn't be hard to throw the initial email with 
> some tweaks from the replies into a FCP so everybody knows the plan, and we 
> have it ratified in case people come along later and 'forget'.

The general intent is that we not lose anything. TRIM is the only
piece of code that really needs resolution upstream causing us to lose
it temporarily. Failure to upstream FreeBSD's TRIM support has been a
recurring source of bugs when integrating new features. As for NFSv4
I've forked the ACL bits in to the OS dependent code - I'll make a
note to make sure that nothing regresses there. This is yet another
example of why there will be a 3 month period for users to try the
port before we cut over.

-M
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: The future of ZFS in FreeBSD

[Warner Losh]

Matt,

This is a fairly comprehensive plan. Kudos for putting it together.

The big question here is do you have a complete list of FreeBSD-specific
changes that will be lost in the cut-over? We've heard about TRIM support
and maybe NFSv4, but are there others that can be identified?

Once you have that list, it wouldn't be hard to throw the initial email
with some tweaks from the replies into a FCP so everybody knows the plan,
and we have it ratified in case people come along later and 'forget'.

Warner
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: The future of ZFS in FreeBSD

[Dennis Glatting]

On Thu, 2018-12-20 at 08:15 -0800, Cy Schubert wrote:
> I also think people will not accept regressions or other POLA
> violations.

+1



___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

RE: The future of ZFS in FreeBSD

[Cy Schubert]

I suspect the idea is to contribute the FreeBSD bits back to ZoL.

My question is, how much of a base does ZoL really have? The enterprise players 
with any significant market share won't touch it -- Ubuntu has but IMO they're 
a niche player (who has violated GPL by including CDDL in their kernel.

The next step should be throwing up a wiki page with a features todo list. 
Also, this is a large enough project to have its own project branch, with the 
goal of having it in head before 13 goes GA. I also think people will not 
accept regressions or other POLA violations.

---
Sent using a tiny phone keyboard.
Apologies for any typos and autocorrect.
Also, this old phone only supports top post. Apologies.

Cy Schubert
 or 
The need of the many outweighs the greed of the few.
---

-Original Message-
From: Alexey Dokuchaev
Sent: 20/12/2018 06:40
To: Eugene M. Zheganin
Cc: freebsd-current@freebsd.org
Subject: Re: The future of ZFS in FreeBSD

On Thu, Dec 20, 2018 at 03:49:38PM +0500, Eugene M. Zheganin wrote:
> On 19.12.2018 23:32, Allan Jude wrote:
> > The biggest thing to remember is that this is still OpenZFS, and still
> > run by the same developers as it has been. We are just commonizing on
> > the repo that has the most features integrated into it.
> 
> Does it mean that ZoF and thus FreeBSD will lose NFSv4 ACLs because
> there is no such thing in ZoL?

+1.  I'm also worried if this would bring more Linuxish bits into our
kernel (cf. LinuxKPI).  Also, I thought that ZFS was never really native
to Linux but implemented through SPL (Solaris Porting Layer), and Linux'
VFS is not ARC-aware unlike Solaris and FreeBSD.

It would be quite upsetting to see ZFS as we know it in FreeBSD become
pessimized because of those things. :-(

./danfe
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

RE: The future of ZFS in FreeBSD

[Cy Schubert]

We would port the feature ourselves and push the code upstream.

---
Sent using a tiny phone keyboard.
Apologies for any typos and autocorrect.
Also, this old phone only supports top post. Apologies.

Cy Schubert
 or 
The need of the many outweighs the greed of the few.
---

-Original Message-
From: Eugene M. Zheganin
Sent: 20/12/2018 03:12
To: freebsd-current@freebsd.org
Subject: Re: The future of ZFS in FreeBSD

Hello,

On 19.12.2018 23:32, Allan Jude wrote:
> The biggest thing to remember is that this is still OpenZFS, and still
> run by the same developers as it has been. We are just commonizing on
> the repo that has the most features integrated into it.

Does it mean that ZoF and thus FreeBSD will lose NFSv4 ACLs because 
there is no such thing in ZoL ?

Eugene.

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: The future of ZFS in FreeBSD

[Alexey Dokuchaev]

On Thu, Dec 20, 2018 at 03:49:38PM +0500, Eugene M. Zheganin wrote:
> On 19.12.2018 23:32, Allan Jude wrote:
> > The biggest thing to remember is that this is still OpenZFS, and still
> > run by the same developers as it has been. We are just commonizing on
> > the repo that has the most features integrated into it.
> 
> Does it mean that ZoF and thus FreeBSD will lose NFSv4 ACLs because
> there is no such thing in ZoL?

+1.  I'm also worried if this would bring more Linuxish bits into our
kernel (cf. LinuxKPI).  Also, I thought that ZFS was never really native
to Linux but implemented through SPL (Solaris Porting Layer), and Linux'
VFS is not ARC-aware unlike Solaris and FreeBSD.

It would be quite upsetting to see ZFS as we know it in FreeBSD become
pessimized because of those things. :-(

./danfe
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: The future of ZFS in FreeBSD

[rb]

> On 20 Dec 2018, at 11:58, Steven Hartland  wrote:
> 
> 
> 
> On 20/12/2018 11:03, Bob Bishop wrote:
>> Hi,
>> 
>>> On 19 Dec 2018, at 23:16, Matthew Macy  wrote:
>>> 
>>> On Wed, Dec 19, 2018 at 15:11 Steven Hartland 
>>> wrote:
>>> 
 Sorry been off for a few weeks so must have missed that, please do prod me
 on again if you don’t see any response to anything not just this. Like many
 others I get so may emails across so many lists it’s more than likely I
 just missed it.
 
 That said would you say that with the right support we can make progress
 on the this prior to the port? I have to ask as the alternative version has
 been on the cusp for many years now so it’s feels more like a distant
 memory than something that may happen, no disrespect to anyone involved, as
 I know all too well how hard it can be to get something like this over the
 line, especially when people have competing priorities.
 
>>> I am hoping that it's sufficiently important to FreeBSD ZFS developers that
>>> they'll give the PR the attention it needs so that it can be merged before
>>> summer. My understanding is that it's mostly suffered from neglect. TRIM is
>>> most important to FreeBSD and it already had its own implementation.
>>> 
>>> https://github.com/zfsonlinux/zfs/pull/5925
>> Please correct me if I’m wrong but this looks a lot less mature than 
>> FreeBSD’s existing TRIM support for ZFS which we’ve had in production for 
>> six years.
>> 
>> What is the rationale here? I’m concerned that it looks like an opportunity 
>> for mighty regressions.
>> 
> This is the case, but overall this solution is thought to be a better 
> approach.
> 
> With anything like this there is always a risk, so we all need a concerted 
> effort to get to one solution.

Not sure what I can contribute, but I can certainly put a box up for testing 
when there’s something to test.

> Regards
> Steve
> 

--
Bob Bishop
r...@gid.co.uk




___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: The future of ZFS in FreeBSD

[Steven Hartland]

On 20/12/2018 11:03, Bob Bishop wrote:

Hi,


On 19 Dec 2018, at 23:16, Matthew Macy  wrote:

On Wed, Dec 19, 2018 at 15:11 Steven Hartland 
wrote:


Sorry been off for a few weeks so must have missed that, please do prod me
on again if you don’t see any response to anything not just this. Like many
others I get so may emails across so many lists it’s more than likely I
just missed it.

That said would you say that with the right support we can make progress
on the this prior to the port? I have to ask as the alternative version has
been on the cusp for many years now so it’s feels more like a distant
memory than something that may happen, no disrespect to anyone involved, as
I know all too well how hard it can be to get something like this over the
line, especially when people have competing priorities.


I am hoping that it's sufficiently important to FreeBSD ZFS developers that
they'll give the PR the attention it needs so that it can be merged before
summer. My understanding is that it's mostly suffered from neglect. TRIM is
most important to FreeBSD and it already had its own implementation.

https://github.com/zfsonlinux/zfs/pull/5925

Please correct me if I’m wrong but this looks a lot less mature than FreeBSD’s 
existing TRIM support for ZFS which we’ve had in production for six years.

What is the rationale here? I’m concerned that it looks like an opportunity for 
mighty regressions.

This is the case, but overall this solution is thought to be a better 
approach.


With anything like this there is always a risk, so we all need a 
concerted effort to get to one solution.


    Regards
    Steve
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: The future of ZFS in FreeBSD

[Eugene M. Zheganin]

Hello,

On 19.12.2018 23:32, Allan Jude wrote:

The biggest thing to remember is that this is still OpenZFS, and still
run by the same developers as it has been. We are just commonizing on
the repo that has the most features integrated into it.


Does it mean that ZoF and thus FreeBSD will lose NFSv4 ACLs because 
there is no such thing in ZoL ?


Eugene.

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: The future of ZFS in FreeBSD

[Bob Bishop]

Hi,

> On 19 Dec 2018, at 23:16, Matthew Macy  wrote:
> 
> On Wed, Dec 19, 2018 at 15:11 Steven Hartland 
> wrote:
> 
>> Sorry been off for a few weeks so must have missed that, please do prod me
>> on again if you don’t see any response to anything not just this. Like many
>> others I get so may emails across so many lists it’s more than likely I
>> just missed it.
>> 
>> That said would you say that with the right support we can make progress
>> on the this prior to the port? I have to ask as the alternative version has
>> been on the cusp for many years now so it’s feels more like a distant
>> memory than something that may happen, no disrespect to anyone involved, as
>> I know all too well how hard it can be to get something like this over the
>> line, especially when people have competing priorities.
>> 
> 
> I am hoping that it's sufficiently important to FreeBSD ZFS developers that
> they'll give the PR the attention it needs so that it can be merged before
> summer. My understanding is that it's mostly suffered from neglect. TRIM is
> most important to FreeBSD and it already had its own implementation.
> 
> https://github.com/zfsonlinux/zfs/pull/5925

Please correct me if I’m wrong but this looks a lot less mature than FreeBSD’s 
existing TRIM support for ZFS which we’ve had in production for six years.

What is the rationale here? I’m concerned that it looks like an opportunity for 
mighty regressions.

> I forwarded you the private communication again as well.
> 
> -M
> 
> 
>> 
>> On Wed, 19 Dec 2018 at 22:52, Matthew Macy  wrote:
>> 
>>> 
>>> 
>>> On Wed, Dec 19, 2018 at 14:47 Steven Hartland 
>>> wrote:
>>> 
 Thanks for the write up most appreciated. One of the more meaty
 differences is that FreeBSD ZFS still has the only merged and production
 ready TRIM support so my question would be are their any plans to address
 this before creating the new port as going back to a world without TRIM
 support wouldn’t be something I’d look forward to.
 
>>> 
>>> Well, then please follow up on the request I CC'd you on a week ago
>>> asking that you engage on the deadlist based TRIM  PR. That's a better
>>> forum for discussing these details than lamenting in public lists.
>>> 
>>> Thanks.
>>> 
>>> -M
>>> 
>>> 
>>> 
>>> 
 On Wed, 19 Dec 2018 at 06:51, Matthew Macy  wrote:
 
> The sources for FreeBSD's ZFS support are currently taken directly
> from Illumos with local ifdefs to support the peculiarities of FreeBSD
> where the Solaris Portability Layer (SPL) shims fall short. FreeBSD
> has regularly pulled changes from Illumos and tried to push back any
> bug fixes and new features done in the context of FreeBSD. In the past
> few years the vast majority of new development in ZFS has taken place
> in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix announced
> that they will be moving to ZoL
> https://www.delphix.com/blog/kickoff-future-eko-2018 This shift means
> that there will be little to no net new development of Illumos. While
> working through the git history of ZoL I have also discovered that
> many races and locking bugs have been fixed in ZoL and never made it
> back to Illumos and thus FreeBSD. This state of affairs has led to a
> general agreement among the stakeholders that I have spoken to that it
> makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf
> has graciously encouraged me to add FreeBSD support directly to ZoL
> https://github.com/zfsonfreebsd/ZoF so that we might all have a single
> shared code base.
> 
> A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port
> Before it can be committed some additional functionality needs to be
> added to the FreeBSD opencrypto framework. These can be found at
> https://reviews.freebsd.org/D18520
> 
> This port will provide FreeBSD users with multi modifier protection,
> project quotas, encrypted datasets, allocation classes, vectorized
> raidz, vectorized checksums, and various command line improvements.
> 
> Before ZoF can be merged back in to ZoL several steps need to be taken:
> - Integrate FreeBSD support into ZoL CI
> - Have most of the ZFS test suite passing
> - Complete additional QA testing at iX
> 
> We at iX Systems need to port ZoL's EC2 CI scripts to work with
> FreeBSD and make sure that most of the ZFS Test Suite (ZTS) passes.
> Being integrated in to their CI will mean that, among other things,
> most integration issues will be caught before a PR is merged upstream
> rather than many months later when it is MFVed into FreeBSD. I’m
> hoping to submit the PR to ZoL some time in January.
> 
> This port will make it easy for end users on a range of releases to
> run the latest version of ZFS. Nonetheless, transitioning away from an
> Illumos based ZFS is not likely to be entirely seamless. The
> 

Re: The future of ZFS in FreeBSD

[Matthew Macy]

On Wed, Dec 19, 2018 at 15:11 Steven Hartland 
wrote:

> Sorry been off for a few weeks so must have missed that, please do prod me
> on again if you don’t see any response to anything not just this. Like many
> others I get so may emails across so many lists it’s more than likely I
> just missed it.
>
> That said would you say that with the right support we can make progress
> on the this prior to the port? I have to ask as the alternative version has
> been on the cusp for many years now so it’s feels more like a distant
> memory than something that may happen, no disrespect to anyone involved, as
> I know all too well how hard it can be to get something like this over the
> line, especially when people have competing priorities.
>

I am hoping that it's sufficiently important to FreeBSD ZFS developers that
they'll give the PR the attention it needs so that it can be merged before
summer. My understanding is that it's mostly suffered from neglect. TRIM is
most important to FreeBSD and it already had its own implementation.

https://github.com/zfsonlinux/zfs/pull/5925

I forwarded you the private communication again as well.

-M


>
> On Wed, 19 Dec 2018 at 22:52, Matthew Macy  wrote:
>
>>
>>
>> On Wed, Dec 19, 2018 at 14:47 Steven Hartland 
>> wrote:
>>
>>> Thanks for the write up most appreciated. One of the more meaty
>>> differences is that FreeBSD ZFS still has the only merged and production
>>> ready TRIM support so my question would be are their any plans to address
>>> this before creating the new port as going back to a world without TRIM
>>> support wouldn’t be something I’d look forward to.
>>>
>>
>> Well, then please follow up on the request I CC'd you on a week ago
>> asking that you engage on the deadlist based TRIM  PR. That's a better
>> forum for discussing these details than lamenting in public lists.
>>
>> Thanks.
>>
>> -M
>>
>>
>>
>>
>>> On Wed, 19 Dec 2018 at 06:51, Matthew Macy  wrote:
>>>
 The sources for FreeBSD's ZFS support are currently taken directly
 from Illumos with local ifdefs to support the peculiarities of FreeBSD
 where the Solaris Portability Layer (SPL) shims fall short. FreeBSD
 has regularly pulled changes from Illumos and tried to push back any
 bug fixes and new features done in the context of FreeBSD. In the past
 few years the vast majority of new development in ZFS has taken place
 in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix announced
 that they will be moving to ZoL
 https://www.delphix.com/blog/kickoff-future-eko-2018 This shift means
 that there will be little to no net new development of Illumos. While
 working through the git history of ZoL I have also discovered that
 many races and locking bugs have been fixed in ZoL and never made it
 back to Illumos and thus FreeBSD. This state of affairs has led to a
 general agreement among the stakeholders that I have spoken to that it
 makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf
 has graciously encouraged me to add FreeBSD support directly to ZoL
 https://github.com/zfsonfreebsd/ZoF so that we might all have a single
 shared code base.

 A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port
 Before it can be committed some additional functionality needs to be
 added to the FreeBSD opencrypto framework. These can be found at
 https://reviews.freebsd.org/D18520

 This port will provide FreeBSD users with multi modifier protection,
 project quotas, encrypted datasets, allocation classes, vectorized
 raidz, vectorized checksums, and various command line improvements.

 Before ZoF can be merged back in to ZoL several steps need to be taken:
 - Integrate FreeBSD support into ZoL CI
 - Have most of the ZFS test suite passing
 - Complete additional QA testing at iX

 We at iX Systems need to port ZoL's EC2 CI scripts to work with
 FreeBSD and make sure that most of the ZFS Test Suite (ZTS) passes.
 Being integrated in to their CI will mean that, among other things,
 most integration issues will be caught before a PR is merged upstream
 rather than many months later when it is MFVed into FreeBSD. I’m
 hoping to submit the PR to ZoL some time in January.

 This port will make it easy for end users on a range of releases to
 run the latest version of ZFS. Nonetheless, transitioning away from an
 Illumos based ZFS is not likely to be entirely seamless. The
 stakeholders I’ve spoken to all agree that this is the best path
 forward but some degree of effort needs to be made to accommodate
 downstream consumers. The current plan is to import ZoF and unhook the
 older Illumos based sources from the build on April 15th or two months
 after iX systems QA deems ZoF stable - which ever comes later. The
 Illumos based sources will be removed some time later - but well
 before 13. This will 

Re: The future of ZFS in FreeBSD

[Steven Hartland]

Sorry been off for a few weeks so must have missed that, please do prod me
on again if you don’t see any response to anything not just this. Like many
others I get so may emails across so many lists it’s more than likely I
just missed it.

That said would you say that with the right support we can make progress on
the this prior to the port? I have to ask as the alternative version has
been on the cusp for many years now so it’s feels more like a distant
memory than something that may happen, no disrespect to anyone involved, as
I know all too well how hard it can be to get something like this over the
line, especially when people have competing priorities.


On Wed, 19 Dec 2018 at 22:52, Matthew Macy  wrote:

>
>
> On Wed, Dec 19, 2018 at 14:47 Steven Hartland 
> wrote:
>
>> Thanks for the write up most appreciated. One of the more meaty
>> differences is that FreeBSD ZFS still has the only merged and production
>> ready TRIM support so my question would be are their any plans to address
>> this before creating the new port as going back to a world without TRIM
>> support wouldn’t be something I’d look forward to.
>>
>
> Well, then please follow up on the request I CC'd you on a week ago asking
> that you engage on the deadlist based TRIM  PR. That's a better forum for
> discussing these details than lamenting in public lists.
>
> Thanks.
>
> -M
>
>
>
>
>> On Wed, 19 Dec 2018 at 06:51, Matthew Macy  wrote:
>>
>>> The sources for FreeBSD's ZFS support are currently taken directly
>>> from Illumos with local ifdefs to support the peculiarities of FreeBSD
>>> where the Solaris Portability Layer (SPL) shims fall short. FreeBSD
>>> has regularly pulled changes from Illumos and tried to push back any
>>> bug fixes and new features done in the context of FreeBSD. In the past
>>> few years the vast majority of new development in ZFS has taken place
>>> in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix announced
>>> that they will be moving to ZoL
>>> https://www.delphix.com/blog/kickoff-future-eko-2018 This shift means
>>> that there will be little to no net new development of Illumos. While
>>> working through the git history of ZoL I have also discovered that
>>> many races and locking bugs have been fixed in ZoL and never made it
>>> back to Illumos and thus FreeBSD. This state of affairs has led to a
>>> general agreement among the stakeholders that I have spoken to that it
>>> makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf
>>> has graciously encouraged me to add FreeBSD support directly to ZoL
>>> https://github.com/zfsonfreebsd/ZoF so that we might all have a single
>>> shared code base.
>>>
>>> A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port
>>> Before it can be committed some additional functionality needs to be
>>> added to the FreeBSD opencrypto framework. These can be found at
>>> https://reviews.freebsd.org/D18520
>>>
>>> This port will provide FreeBSD users with multi modifier protection,
>>> project quotas, encrypted datasets, allocation classes, vectorized
>>> raidz, vectorized checksums, and various command line improvements.
>>>
>>> Before ZoF can be merged back in to ZoL several steps need to be taken:
>>> - Integrate FreeBSD support into ZoL CI
>>> - Have most of the ZFS test suite passing
>>> - Complete additional QA testing at iX
>>>
>>> We at iX Systems need to port ZoL's EC2 CI scripts to work with
>>> FreeBSD and make sure that most of the ZFS Test Suite (ZTS) passes.
>>> Being integrated in to their CI will mean that, among other things,
>>> most integration issues will be caught before a PR is merged upstream
>>> rather than many months later when it is MFVed into FreeBSD. I’m
>>> hoping to submit the PR to ZoL some time in January.
>>>
>>> This port will make it easy for end users on a range of releases to
>>> run the latest version of ZFS. Nonetheless, transitioning away from an
>>> Illumos based ZFS is not likely to be entirely seamless. The
>>> stakeholders I’ve spoken to all agree that this is the best path
>>> forward but some degree of effort needs to be made to accommodate
>>> downstream consumers. The current plan is to import ZoF and unhook the
>>> older Illumos based sources from the build on April 15th or two months
>>> after iX systems QA deems ZoF stable - which ever comes later. The
>>> Illumos based sources will be removed some time later - but well
>>> before 13. This will give users a 3 month period during which both the
>>> port and legacy Illumos based ZFS will be available to users. Pools
>>> should interoperate between ZoF and legacy provided the user does not
>>> enable any features available only in ZoF. We will try to accommodate
>>> any downstream consumers in the event that they need that date pushed
>>> back. We ask that any downstream consumers who are particularly
>>> sensitive to changes start testing the port when it is formally
>>> announced and report back any issues they have. I will do my best to
>>> 

Re: The future of ZFS in FreeBSD

[Matthew Macy]

On Wed, Dec 19, 2018 at 14:47 Steven Hartland 
wrote:

> Thanks for the write up most appreciated. One of the more meaty
> differences is that FreeBSD ZFS still has the only merged and production
> ready TRIM support so my question would be are their any plans to address
> this before creating the new port as going back to a world without TRIM
> support wouldn’t be something I’d look forward to.
>

Well, then please follow up on the request I CC'd you on a week ago asking
that you engage on the deadlist based TRIM  PR. That's a better forum for
discussing these details than lamenting in public lists.

Thanks.

-M




> On Wed, 19 Dec 2018 at 06:51, Matthew Macy  wrote:
>
>> The sources for FreeBSD's ZFS support are currently taken directly
>> from Illumos with local ifdefs to support the peculiarities of FreeBSD
>> where the Solaris Portability Layer (SPL) shims fall short. FreeBSD
>> has regularly pulled changes from Illumos and tried to push back any
>> bug fixes and new features done in the context of FreeBSD. In the past
>> few years the vast majority of new development in ZFS has taken place
>> in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix announced
>> that they will be moving to ZoL
>> https://www.delphix.com/blog/kickoff-future-eko-2018 This shift means
>> that there will be little to no net new development of Illumos. While
>> working through the git history of ZoL I have also discovered that
>> many races and locking bugs have been fixed in ZoL and never made it
>> back to Illumos and thus FreeBSD. This state of affairs has led to a
>> general agreement among the stakeholders that I have spoken to that it
>> makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf
>> has graciously encouraged me to add FreeBSD support directly to ZoL
>> https://github.com/zfsonfreebsd/ZoF so that we might all have a single
>> shared code base.
>>
>> A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port
>> Before it can be committed some additional functionality needs to be
>> added to the FreeBSD opencrypto framework. These can be found at
>> https://reviews.freebsd.org/D18520
>>
>> This port will provide FreeBSD users with multi modifier protection,
>> project quotas, encrypted datasets, allocation classes, vectorized
>> raidz, vectorized checksums, and various command line improvements.
>>
>> Before ZoF can be merged back in to ZoL several steps need to be taken:
>> - Integrate FreeBSD support into ZoL CI
>> - Have most of the ZFS test suite passing
>> - Complete additional QA testing at iX
>>
>> We at iX Systems need to port ZoL's EC2 CI scripts to work with
>> FreeBSD and make sure that most of the ZFS Test Suite (ZTS) passes.
>> Being integrated in to their CI will mean that, among other things,
>> most integration issues will be caught before a PR is merged upstream
>> rather than many months later when it is MFVed into FreeBSD. I’m
>> hoping to submit the PR to ZoL some time in January.
>>
>> This port will make it easy for end users on a range of releases to
>> run the latest version of ZFS. Nonetheless, transitioning away from an
>> Illumos based ZFS is not likely to be entirely seamless. The
>> stakeholders I’ve spoken to all agree that this is the best path
>> forward but some degree of effort needs to be made to accommodate
>> downstream consumers. The current plan is to import ZoF and unhook the
>> older Illumos based sources from the build on April 15th or two months
>> after iX systems QA deems ZoF stable - which ever comes later. The
>> Illumos based sources will be removed some time later - but well
>> before 13. This will give users a 3 month period during which both the
>> port and legacy Illumos based ZFS will be available to users. Pools
>> should interoperate between ZoF and legacy provided the user does not
>> enable any features available only in ZoF. We will try to accommodate
>> any downstream consumers in the event that they need that date pushed
>> back. We ask that any downstream consumers who are particularly
>> sensitive to changes start testing the port when it is formally
>> announced and report back any issues they have. I will do my best to
>> ensure that this message is communicated to all those who it may
>> concern. However, I can’t ensure that everyone reads these lists. That
>> is the responsibility of -CURRENT users.
>>
>> -M
>>
> ___
>> freebsd...@freebsd.org mailing list
>> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
>> To unsubscribe, send any mail to "freebsd-fs-unsubscr...@freebsd.org"
>>
>
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: The future of ZFS in FreeBSD

[Steven Hartland]

Thanks for the write up most appreciated. One of the more meaty differences
is that FreeBSD ZFS still has the only merged and production ready TRIM
support so my question would be are their any plans to address this before
creating the new port as going back to a world without TRIM support
wouldn’t be something I’d look forward to.

On Wed, 19 Dec 2018 at 06:51, Matthew Macy  wrote:

> The sources for FreeBSD's ZFS support are currently taken directly
> from Illumos with local ifdefs to support the peculiarities of FreeBSD
> where the Solaris Portability Layer (SPL) shims fall short. FreeBSD
> has regularly pulled changes from Illumos and tried to push back any
> bug fixes and new features done in the context of FreeBSD. In the past
> few years the vast majority of new development in ZFS has taken place
> in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix announced
> that they will be moving to ZoL
> https://www.delphix.com/blog/kickoff-future-eko-2018 This shift means
> that there will be little to no net new development of Illumos. While
> working through the git history of ZoL I have also discovered that
> many races and locking bugs have been fixed in ZoL and never made it
> back to Illumos and thus FreeBSD. This state of affairs has led to a
> general agreement among the stakeholders that I have spoken to that it
> makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf
> has graciously encouraged me to add FreeBSD support directly to ZoL
> https://github.com/zfsonfreebsd/ZoF so that we might all have a single
> shared code base.
>
> A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port
> Before it can be committed some additional functionality needs to be
> added to the FreeBSD opencrypto framework. These can be found at
> https://reviews.freebsd.org/D18520
>
> This port will provide FreeBSD users with multi modifier protection,
> project quotas, encrypted datasets, allocation classes, vectorized
> raidz, vectorized checksums, and various command line improvements.
>
> Before ZoF can be merged back in to ZoL several steps need to be taken:
> - Integrate FreeBSD support into ZoL CI
> - Have most of the ZFS test suite passing
> - Complete additional QA testing at iX
>
> We at iX Systems need to port ZoL's EC2 CI scripts to work with
> FreeBSD and make sure that most of the ZFS Test Suite (ZTS) passes.
> Being integrated in to their CI will mean that, among other things,
> most integration issues will be caught before a PR is merged upstream
> rather than many months later when it is MFVed into FreeBSD. I’m
> hoping to submit the PR to ZoL some time in January.
>
> This port will make it easy for end users on a range of releases to
> run the latest version of ZFS. Nonetheless, transitioning away from an
> Illumos based ZFS is not likely to be entirely seamless. The
> stakeholders I’ve spoken to all agree that this is the best path
> forward but some degree of effort needs to be made to accommodate
> downstream consumers. The current plan is to import ZoF and unhook the
> older Illumos based sources from the build on April 15th or two months
> after iX systems QA deems ZoF stable - which ever comes later. The
> Illumos based sources will be removed some time later - but well
> before 13. This will give users a 3 month period during which both the
> port and legacy Illumos based ZFS will be available to users. Pools
> should interoperate between ZoF and legacy provided the user does not
> enable any features available only in ZoF. We will try to accommodate
> any downstream consumers in the event that they need that date pushed
> back. We ask that any downstream consumers who are particularly
> sensitive to changes start testing the port when it is formally
> announced and report back any issues they have. I will do my best to
> ensure that this message is communicated to all those who it may
> concern. However, I can’t ensure that everyone reads these lists. That
> is the responsibility of -CURRENT users.
>
> -M
> ___
> freebsd...@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-fs
> To unsubscribe, send any mail to "freebsd-fs-unsubscr...@freebsd.org"
>
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: The future of ZFS in FreeBSD

[Pete Wright]

On 12/19/18 10:32 AM, Allan Jude wrote:


The biggest thing to remember is that this is still OpenZFS, and still
run by the same developers as it has been. We are just commonizing on
the repo that has the most features integrated into it.


thanks for the clarification on this, as i was a bit confused as well. :)

One question - is there a plan to rename things to not be linux 
specific, especially since it seems that this effort will result in the 
original goal of the OpenZFS repro (if I am reading things correctly)?  
Or is that not going to happen because ZoL has the most commercial backing?


It's probably a mute point - but I do take pride in the fact that 
FreeBSD has had support ZFS for so long and has allowed me to do things 
for my customers that I can't do on Linux...so I guess what I'm saying 
is I hope we don't loose credit for the hard work we've put into getting 
ZFS mainstream over the years :)


Thanks!
-pete


--
Pete Wright
p...@nomadlogic.org
@nomadlogicLA

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: The future of ZFS in FreeBSD

[Matthew Ahrens]

On Wed, Dec 19, 2018 at 8:35 AM Shawn Webb 
wrote:

> I'm curious what this means for OpenZFS.
>

OpenZFS will continue to be an umbrella project for coordinating all work
on open-source ZFS.  The primary activities of OpenZFS are the annual OpenZFS
Developer Summit
 and the
monthly OpenZFS Leadership Meetings

.
There is also an OpenZFS GitHub repo, which has identical code to illumos.
This is used to streamline the process of getting changes into illumos
(removing some of the burden of testing and the illumos RTI process).  This
repo will continue to exist as long as there's need for it and folks able
to keep the automation running.  The naming of this GitHub project was
perhaps unfortunate, since it is specific to illumos and not
platform-agnostic.

 I was under the impression that OpenZFS was the upstream for all the ZFS
> implementations (sans
> Oracle).


Being "upstream" is more a matter of degree than an absolute ordering.  In
the past, most ZFS features have originated in illumos, and those changes
have been ported to the other platforms (notably FreeBSD and ZoL).  Looking
forward, we see more features originating in ZoL.  The OpenZFS community
(including FreeBSD) is working to put in place technical processes to
ensure that new ZFS features are available on all platforms.  The original
post in this thread was a proposal for how to do that.

--matt
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: The future of ZFS in FreeBSD

[Allan Jude]

On 2018-12-19 11:30, Shawn Webb wrote:
> On Tue, Dec 18, 2018 at 10:49:48PM -0800, Matthew Macy wrote:
>> The sources for FreeBSD's ZFS support are currently taken directly
>> from Illumos with local ifdefs to support the peculiarities of FreeBSD
>> where the Solaris Portability Layer (SPL) shims fall short. FreeBSD
>> has regularly pulled changes from Illumos and tried to push back any
>> bug fixes and new features done in the context of FreeBSD. In the past
>> few years the vast majority of new development in ZFS has taken place
>> in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix announced
>> that they will be moving to ZoL
>> https://www.delphix.com/blog/kickoff-future-eko-2018 This shift means
>> that there will be little to no net new development of Illumos. While
>> working through the git history of ZoL I have also discovered that
>> many races and locking bugs have been fixed in ZoL and never made it
>> back to Illumos and thus FreeBSD. This state of affairs has led to a
>> general agreement among the stakeholders that I have spoken to that it
>> makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf
>> has graciously encouraged me to add FreeBSD support directly to ZoL
>> https://github.com/zfsonfreebsd/ZoF so that we might all have a single
>> shared code base.
>>
>> A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port
>> Before it can be committed some additional functionality needs to be
>> added to the FreeBSD opencrypto framework. These can be found at
>> https://reviews.freebsd.org/D18520
>>
>> This port will provide FreeBSD users with multi modifier protection,
>> project quotas, encrypted datasets, allocation classes, vectorized
>> raidz, vectorized checksums, and various command line improvements.
>>
>> Before ZoF can be merged back in to ZoL several steps need to be taken:
>> - Integrate FreeBSD support into ZoL CI
>> - Have most of the ZFS test suite passing
>> - Complete additional QA testing at iX
>>
>> We at iX Systems need to port ZoL's EC2 CI scripts to work with
>> FreeBSD and make sure that most of the ZFS Test Suite (ZTS) passes.
>> Being integrated in to their CI will mean that, among other things,
>> most integration issues will be caught before a PR is merged upstream
>> rather than many months later when it is MFVed into FreeBSD. I???m
>> hoping to submit the PR to ZoL some time in January.
>>
>> This port will make it easy for end users on a range of releases to
>> run the latest version of ZFS. Nonetheless, transitioning away from an
>> Illumos based ZFS is not likely to be entirely seamless. The
>> stakeholders I???ve spoken to all agree that this is the best path
>> forward but some degree of effort needs to be made to accommodate
>> downstream consumers. The current plan is to import ZoF and unhook the
>> older Illumos based sources from the build on April 15th or two months
>> after iX systems QA deems ZoF stable - which ever comes later. The
>> Illumos based sources will be removed some time later - but well
>> before 13. This will give users a 3 month period during which both the
>> port and legacy Illumos based ZFS will be available to users. Pools
>> should interoperate between ZoF and legacy provided the user does not
>> enable any features available only in ZoF. We will try to accommodate
>> any downstream consumers in the event that they need that date pushed
>> back. We ask that any downstream consumers who are particularly
>> sensitive to changes start testing the port when it is formally
>> announced and report back any issues they have. I will do my best to
>> ensure that this message is communicated to all those who it may
>> concern. However, I can???t ensure that everyone reads these lists. That
>> is the responsibility of -CURRENT users.
> 
> Hey Matt,
> 
> Thank you for your detailed and informative post. It really helps
> downstream consumers of FreeBSD.
> 
> I'm curious what this means for OpenZFS. I was under the impression
> that OpenZFS was the upstream for all the ZFS implementations (sans
> Oracle).
> 
> Thanks,
> 

The original goal of the OpenZFS project was to have a single OS
agnostic repo where all of the common code lives, to try to make it
easier to have all of the implementations be relatively in sync with
each other, and have feature parity and import compatibility.

For the last 5 years, this goal is been out of reach, because no one
could justify the effort and expense of maintaining the 'one true repo'
when no one would actually be using or benefiting from that repo directly.

Today, the OpenZFS repo is just a fork of the illumos-gate repo, but
where pull requests are accepted, and where previous Delphix employees
would deal with the process of trying to upstream patches to illumos.
This process has not worked well recently, as things have gotten stuck
waiting for 'merge advocates' in illumos.

The ZoF effort will move us closer to the goal of a common repo. The
general idea will be to have 'OS 

Re: The future of ZFS in FreeBSD

[Andrew Turner]

> On 19 Dec 2018, at 08:35, Matthew Macy  wrote:
> 
> On Tue, Dec 18, 2018 at 11:49 PM Enji Cooper  > wrote:
>> 
>> Hello Matthew,
>> 
>> I appreciate the long write up, as someone who still uses FreeBSD ZFS on my 
>> NAS box and knowing some of the history with ZFS on *Solaris, etc. Something 
>> like this was bound to happen with post the Oracle buyout.
>> 
>>> On Dec 18, 2018, at 10:49 PM, Matthew Macy  wrote:
>>> 
>>> The sources for FreeBSD's ZFS support are currently taken directly
>>> from Illumos with local ifdefs to support the peculiarities of FreeBSD
>>> where the Solaris Portability Layer (SPL) shims fall short. FreeBSD
>>> has regularly pulled changes from Illumos and tried to push back any
>>> bug fixes and new features done in the context of FreeBSD. In the past
>>> few years the vast majority of new development in ZFS has taken place
>>> in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix announced
>>> that they will be moving to ZoL
>>> https://www.delphix.com/blog/kickoff-future-eko-2018 This shift means
>>> that there will be little to no net new development of Illumos. While
>>> working through the git history of ZoL I have also discovered that
>>> many races and locking bugs have been fixed in ZoL and never made it
>>> back to Illumos and thus FreeBSD. This state of affairs has led to a
>>> general agreement among the stakeholders that I have spoken to that it
>>> makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf
>>> has graciously encouraged me to add FreeBSD support directly to ZoL
>>> https://github.com/zfsonfreebsd/ZoF so that we might all have a single
>>> shared code base.
>>> 
>>> A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port
>>> Before it can be committed some additional functionality needs to be
>>> added to the FreeBSD opencrypto framework. These can be found at
>>> https://reviews.freebsd.org/D18520
>>> 
>>> This port will provide FreeBSD users with multi modifier protection,
>>> project quotas, encrypted datasets, allocation classes, vectorized
>>> raidz, vectorized checksums, and various command line improvements.
>>> 
>>> Before ZoF can be merged back in to ZoL several steps need to be taken:
>>> - Integrate FreeBSD support into ZoL CI
>>> - Have most of the ZFS test suite passing
>>> - Complete additional QA testing at iX
>> 
>> Can you please describe the testing process that will be employed to verify 
>> the sanity of the ZoL on FreeBSD port? Should other large companies who use 
>> ZFS on FreeBSD (Panzura?) chime in and the ZFS on FreeBSD community (as a 
>> whole) collaborate to better suss out issues with the ZoL port?
> 
> The ZFS test suite itself provides ~80% coverage
> https://codecov.io/gh/zfsonlinux/zfs/branch/master 
>  - FreeBSD currently
> lacks equivalent gcov support, but presumably it would provide
> comparable coverage here. Andrew Turner has some form of kernel gcov
> support that he uses with syzkaller. However, I believe that it isn't
> sufficient for this purpose.

The code I have is to trace the kernel part of a single thread, e.g. what 
happens in the kernel when you make a system call. It can trace function either 
function entry or places in the code with a conditional statement, e.g. an if, 
while, or switch statement. If this is enough for your case a tool could be 
written to track coverage from the tests.

I expect to update the review soon. I’m working on a man page, but have been 
too busy with work and other projects recently to finish writing it.

Andrew

___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: The future of ZFS in FreeBSD

[Shawn Webb]

On Tue, Dec 18, 2018 at 10:49:48PM -0800, Matthew Macy wrote:
> The sources for FreeBSD's ZFS support are currently taken directly
> from Illumos with local ifdefs to support the peculiarities of FreeBSD
> where the Solaris Portability Layer (SPL) shims fall short. FreeBSD
> has regularly pulled changes from Illumos and tried to push back any
> bug fixes and new features done in the context of FreeBSD. In the past
> few years the vast majority of new development in ZFS has taken place
> in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix announced
> that they will be moving to ZoL
> https://www.delphix.com/blog/kickoff-future-eko-2018 This shift means
> that there will be little to no net new development of Illumos. While
> working through the git history of ZoL I have also discovered that
> many races and locking bugs have been fixed in ZoL and never made it
> back to Illumos and thus FreeBSD. This state of affairs has led to a
> general agreement among the stakeholders that I have spoken to that it
> makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf
> has graciously encouraged me to add FreeBSD support directly to ZoL
> https://github.com/zfsonfreebsd/ZoF so that we might all have a single
> shared code base.
> 
> A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port
> Before it can be committed some additional functionality needs to be
> added to the FreeBSD opencrypto framework. These can be found at
> https://reviews.freebsd.org/D18520
> 
> This port will provide FreeBSD users with multi modifier protection,
> project quotas, encrypted datasets, allocation classes, vectorized
> raidz, vectorized checksums, and various command line improvements.
> 
> Before ZoF can be merged back in to ZoL several steps need to be taken:
> - Integrate FreeBSD support into ZoL CI
> - Have most of the ZFS test suite passing
> - Complete additional QA testing at iX
> 
> We at iX Systems need to port ZoL's EC2 CI scripts to work with
> FreeBSD and make sure that most of the ZFS Test Suite (ZTS) passes.
> Being integrated in to their CI will mean that, among other things,
> most integration issues will be caught before a PR is merged upstream
> rather than many months later when it is MFVed into FreeBSD. I???m
> hoping to submit the PR to ZoL some time in January.
> 
> This port will make it easy for end users on a range of releases to
> run the latest version of ZFS. Nonetheless, transitioning away from an
> Illumos based ZFS is not likely to be entirely seamless. The
> stakeholders I???ve spoken to all agree that this is the best path
> forward but some degree of effort needs to be made to accommodate
> downstream consumers. The current plan is to import ZoF and unhook the
> older Illumos based sources from the build on April 15th or two months
> after iX systems QA deems ZoF stable - which ever comes later. The
> Illumos based sources will be removed some time later - but well
> before 13. This will give users a 3 month period during which both the
> port and legacy Illumos based ZFS will be available to users. Pools
> should interoperate between ZoF and legacy provided the user does not
> enable any features available only in ZoF. We will try to accommodate
> any downstream consumers in the event that they need that date pushed
> back. We ask that any downstream consumers who are particularly
> sensitive to changes start testing the port when it is formally
> announced and report back any issues they have. I will do my best to
> ensure that this message is communicated to all those who it may
> concern. However, I can???t ensure that everyone reads these lists. That
> is the responsibility of -CURRENT users.

Hey Matt,

Thank you for your detailed and informative post. It really helps
downstream consumers of FreeBSD.

I'm curious what this means for OpenZFS. I was under the impression
that OpenZFS was the upstream for all the ZFS implementations (sans
Oracle).

Thanks,

-- 
Shawn Webb
Cofounder and Security Engineer
HardenedBSD

Tor-ified Signal:+1 443-546-8752
Tor+XMPP+OTR:latt...@is.a.hacker.sx
GPG Key ID:  0x6A84658F52456EEE
GPG Key Fingerprint: 2ABA B6BD EF6A F486 BE89  3D9E 6A84 658F 5245 6EEE


signature.asc
Description: PGP signature

Re: The future of ZFS in FreeBSD

[Matthew Macy]

On Tue, Dec 18, 2018 at 11:49 PM Enji Cooper  wrote:
>
> Hello Matthew,
>
> I appreciate the long write up, as someone who still uses FreeBSD ZFS on my 
> NAS box and knowing some of the history with ZFS on *Solaris, etc. Something 
> like this was bound to happen with post the Oracle buyout.
>
> > On Dec 18, 2018, at 10:49 PM, Matthew Macy  wrote:
> >
> > The sources for FreeBSD's ZFS support are currently taken directly
> > from Illumos with local ifdefs to support the peculiarities of FreeBSD
> > where the Solaris Portability Layer (SPL) shims fall short. FreeBSD
> > has regularly pulled changes from Illumos and tried to push back any
> > bug fixes and new features done in the context of FreeBSD. In the past
> > few years the vast majority of new development in ZFS has taken place
> > in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix announced
> > that they will be moving to ZoL
> > https://www.delphix.com/blog/kickoff-future-eko-2018 This shift means
> > that there will be little to no net new development of Illumos. While
> > working through the git history of ZoL I have also discovered that
> > many races and locking bugs have been fixed in ZoL and never made it
> > back to Illumos and thus FreeBSD. This state of affairs has led to a
> > general agreement among the stakeholders that I have spoken to that it
> > makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf
> > has graciously encouraged me to add FreeBSD support directly to ZoL
> > https://github.com/zfsonfreebsd/ZoF so that we might all have a single
> > shared code base.
> >
> > A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port
> > Before it can be committed some additional functionality needs to be
> > added to the FreeBSD opencrypto framework. These can be found at
> > https://reviews.freebsd.org/D18520
> >
> > This port will provide FreeBSD users with multi modifier protection,
> > project quotas, encrypted datasets, allocation classes, vectorized
> > raidz, vectorized checksums, and various command line improvements.
> >
> > Before ZoF can be merged back in to ZoL several steps need to be taken:
> > - Integrate FreeBSD support into ZoL CI
> > - Have most of the ZFS test suite passing
> > - Complete additional QA testing at iX
>
> Can you please describe the testing process that will be employed to verify 
> the sanity of the ZoL on FreeBSD port? Should other large companies who use 
> ZFS on FreeBSD (Panzura?) chime in and the ZFS on FreeBSD community (as a 
> whole) collaborate to better suss out issues with the ZoL port?

The ZFS test suite itself provides ~80% coverage
https://codecov.io/gh/zfsonlinux/zfs/branch/master - FreeBSD currently
lacks equivalent gcov support, but presumably it would provide
comparable coverage here. Andrew Turner has some form of kernel gcov
support that he uses with syzkaller. However, I believe that it isn't
sufficient for this purpose. ZoL requires that coverage only increase
over time. If a change would decrease total coverage new tests have to
be included with the change. There is a command called ztest which
runs in user space that quickly covers a large percentage of the code
base where it doesn't integrate directly with geom or VFS (disk access
is emulated with files, kernel threads are emulated with pthreads,
etc). ztest has been broken in HEAD for the past 2 years (it triggers
an assertion failure), whereas it is not broken in ZoF. Joe Maloney is
the head of QA at iX and can likely give you more detail on their
approach to testing. This e-mail was an implicit request to Panzura to
do whatever testing that they can.

>
> > We at iX Systems need to port ZoL's EC2 CI scripts to work with
> > FreeBSD and make sure that most of the ZFS Test Suite (ZTS) passes.
> > Being integrated in to their CI will mean that, among other things,
> > most integration issues will be caught before a PR is merged upstream
> > rather than many months later when it is MFVed into FreeBSD. I’m
> > hoping to submit the PR to ZoL some time in January.
>
> Could you please better describe this, in particular provide pointers to any 
> scripts that might be executed with the ZoL port?

The ZFS test suite is in ZoL under tests, it makes some Linux specific
assumptions about paths to binaries like ksh etc and it appears to
require some additional configuration options. I started fixing things
before I decided to punt and ask that a porter at iX fix it so that I
could focus on areas that make more sense for me to spend my time on.
If it doesn't get handled I will of course go back to it. The CI
scripts are not something I've looked at. My intention is again that
the people who would be responsible for keeping them up at iX would
close the loop. It's close to Christmas time right now and folks are
inevitably preoccupied with family matters. I sent this mail in
advance of resolving these details to give any outside stakeholders a
heads up as far out as possible. "Shadow" stakeholders coming out of

Re: The future of ZFS in FreeBSD

[Enji Cooper]

Hello Matthew,

I appreciate the long write up, as someone who still uses FreeBSD ZFS on my NAS 
box and knowing some of the history with ZFS on *Solaris, etc. Something like 
this was bound to happen with post the Oracle buyout.

> On Dec 18, 2018, at 10:49 PM, Matthew Macy  wrote:
> 
> The sources for FreeBSD's ZFS support are currently taken directly
> from Illumos with local ifdefs to support the peculiarities of FreeBSD
> where the Solaris Portability Layer (SPL) shims fall short. FreeBSD
> has regularly pulled changes from Illumos and tried to push back any
> bug fixes and new features done in the context of FreeBSD. In the past
> few years the vast majority of new development in ZFS has taken place
> in DelphixOS and zfsonlinux (ZoL). Earlier this year Delphix announced
> that they will be moving to ZoL
> https://www.delphix.com/blog/kickoff-future-eko-2018 This shift means
> that there will be little to no net new development of Illumos. While
> working through the git history of ZoL I have also discovered that
> many races and locking bugs have been fixed in ZoL and never made it
> back to Illumos and thus FreeBSD. This state of affairs has led to a
> general agreement among the stakeholders that I have spoken to that it
> makes sense to rebase FreeBSD's ZFS on ZoL. Brian Behlendorf
> has graciously encouraged me to add FreeBSD support directly to ZoL
> https://github.com/zfsonfreebsd/ZoF so that we might all have a single
> shared code base.
> 
> A port for ZoF can be found at https://github.com/miwi-fbsd/zof-port
> Before it can be committed some additional functionality needs to be
> added to the FreeBSD opencrypto framework. These can be found at
> https://reviews.freebsd.org/D18520
> 
> This port will provide FreeBSD users with multi modifier protection,
> project quotas, encrypted datasets, allocation classes, vectorized
> raidz, vectorized checksums, and various command line improvements.
> 
> Before ZoF can be merged back in to ZoL several steps need to be taken:
> - Integrate FreeBSD support into ZoL CI
> - Have most of the ZFS test suite passing
> - Complete additional QA testing at iX

Can you please describe the testing process that will be employed to verify the 
sanity of the ZoL on FreeBSD port? Should other large companies who use ZFS on 
FreeBSD (Panzura?) chime in and the ZFS on FreeBSD community (as a whole) 
collaborate to better suss out issues with the ZoL port?

> We at iX Systems need to port ZoL's EC2 CI scripts to work with
> FreeBSD and make sure that most of the ZFS Test Suite (ZTS) passes.
> Being integrated in to their CI will mean that, among other things,
> most integration issues will be caught before a PR is merged upstream
> rather than many months later when it is MFVed into FreeBSD. I’m
> hoping to submit the PR to ZoL some time in January.

Could you please better describe this, in particular provide pointers to any 
scripts that might be executed with the ZoL port?

> This port will make it easy for end users on a range of releases to
> run the latest version of ZFS. Nonetheless, transitioning away from an
> Illumos based ZFS is not likely to be entirely seamless. The
> stakeholders I’ve spoken to all agree that this is the best path
> forward but some degree of effort needs to be made to accommodate
> downstream consumers. The current plan is to import ZoF and unhook the
> older Illumos based sources from the build on April 15th or two months
> after iX systems QA deems ZoF stable - which ever comes later. The
> Illumos based sources will be removed some time later - but well
> before 13. This will give users a 3 month period during which both the
> port and legacy Illumos based ZFS will be available to users. Pools
> should interoperate between ZoF and legacy provided the user does not
> enable any features available only in ZoF. We will try to accommodate
> any downstream consumers in the event that they need that date pushed
> back. We ask that any downstream consumers who are particularly
> sensitive to changes start testing the port when it is formally
> announced and report back any issues they have. I will do my best to
> ensure that this message is communicated to all those who it may
> concern. However, I can’t ensure that everyone reads these lists. That
> is the responsibility of -CURRENT users.


As a thought: is there a way for the ZoL port to run in parallel with the 
non-ZoL port, kind of like in an audit/read-only/verify-only mode?

Thanks so much,
-Enji


signature.asc
Description: Message signed with OpenPGP