Re: 'fabric1' and 'fabric2' packages
> > Once a package is popular > enough, it will get "promoted" to the main community repo. That should > either never happen since there's not many fabric1 users, or happen > after python3 migration. > > To avoid also making python2-pytest-relaxed and python2-invoke, > python2-paramiko doesn't run a test suite. Unfortunate but made my life > easier. > Thanks for the updates - makes sense. > That's basically it for now, ping me if/when 1.x is working on Python 3? > Keep an eye on the list - I expect most of it to get worked out on here, certainly it will be announced post merge. > Once that's working (as long as it's in the next couple months) I'll: > - Update fabric1 in the AUR > - Delete python2-paramiko from the AUR, since nothing else is using it. > - Talk to the Arch Linux 'fabric' maintainer about renaming fabric -> > fabric2 > - Submit a fabric1 package to Debian and talk to them about renaming > fabric -> fabric2 > Seems sensible - thanks! > P.S. I looked at my fabfile thoroughly over the last few weeks. I have a > (short!) list of things blocking me from upgrading, if you would be > interested in specifics. > As long as you skim the upgrading doc (http://www.fabfile.org/upgrading.html) to see what's already a known issue, by all means. Feel free to send that to just-me if you want to avoid the noise for other folks. -- Jeff Forcier Unix sysadmin; Python engineer http://bitprophet.org
Re: 'fabric1' and 'fabric2' packages
On 2020-05-29 06:14, Jeff Forcier wrote: This class of issue is a longstanding rare-but-never-vanquished frustration with Paramiko. I'd definitely make sure the Paramiko version is a recent (re: list of releases; I have not put any out in the last N months...) release. Well, I was able to reproduce my hang with pure ssh after a bit of work, so it looks like my package is probably working after all, and I just have unrelated issues. I'll debug on my own time. I did try that before and it didn't hang, but it looks like I got freakishly (un)lucky. I could reproduce on every fabric and paramiko version, so I started trying ssh again :). I put together AUR packages in Arch Linux. - https://aur.archlinux.org/packages/fabric1/ - https://aur.archlinux.org/packages/python2-paramiko/ The AUR is open to contributions from randos like me, but doesn't have maintainer support, and you can't directly use the package manager to download and install things. I'd say 20% of my installed packages are from the AUR, this is pretty normal on Arch. Once a package is popular enough, it will get "promoted" to the main community repo. That should either never happen since there's not many fabric1 users, or happen after python3 migration. To avoid also making python2-pytest-relaxed and python2-invoke, python2-paramiko doesn't run a test suite. Unfortunate but made my life easier. That's basically it for now, ping me if/when 1.x is working on Python 3? Once that's working (as long as it's in the next couple months) I'll: - Update fabric1 in the AUR - Delete python2-paramiko from the AUR, since nothing else is using it. - Talk to the Arch Linux 'fabric' maintainer about renaming fabric -> fabric2 - Submit a fabric1 package to Debian and talk to them about renaming fabric -> fabric2 P.S. I looked at my fabfile thoroughly over the last few weeks. I have a (short!) list of things blocking me from upgrading, if you would be interested in specifics.
Re: 'fabric1' and 'fabric2' packages
> > OK, well some small updates. > - Debian has indeed closed the door on new python2 packages, trying to > eliminate python2 from their ship stack altogether > Not unexpected, but thanks anyway for asking them! > - Arch doesn't seem to have any problem with it, but the default install > for everything is python2. That means I'll need to re-add a python2 > version of paramiko as a dependency (and maybe invoke? I don't think > so). > Fabric 1 does not need Invoke for anything but development tasks. Paramiko has not dropped Python 2 yet so the latest versions ought to function fine for this. > I wrote an Arch package, which installs and starts up Fabric 1.14.1 > running on Python 2 fine. However, it looks like there are some bugs in > this version--running it on my production stuff, ssh output hangs > partway through. I suspect some bad interaction with paramiko, or maybe > a buggy paramiko version is being selected. I'd like to fix it, but I'm > feeling a bit lost on tracking down the problem honestly. I'll take a > break and figure out how to get more debug info later. > This class of issue is a longstanding rare-but-never-vanquished frustration with Paramiko. I'd definitely make sure the Paramiko version is a recent (re: list of releases; I have not put any out in the last N months...) release. -- Jeff Forcier Unix sysadmin; Python engineer http://bitprophet.org
Re: 'fabric1' and 'fabric2' packages
- I think it's worth asking the distros if they would allow a Python 2 Fabric 1 option, for those users who are still stuck supporting Py2, but like you say I would not be surprised if they've already closed that door. Will do. I'll hand them a package and they can say yes or no (asking before making a package will get stupid responses, sadly). It's not that much work, and I'll learn what I need for a py3 fabric1 package even if they say no. OK, well some small updates. - Debian has indeed closed the door on new python2 packages, trying to eliminate python2 from their ship stack altogether - Arch doesn't seem to have any problem with it, but the default install for everything is python2. That means I'll need to re-add a python2 version of paramiko as a dependency (and maybe invoke? I don't think so). I wrote an Arch package, which installs and starts up Fabric 1.14.1 running on Python 2 fine. However, it looks like there are some bugs in this version--running it on my production stuff, ssh output hangs partway through. I suspect some bad interaction with paramiko, or maybe a buggy paramiko version is being selected. I'd like to fix it, but I'm feeling a bit lost on tracking down the problem honestly. I'll take a break and figure out how to get more debug info later.
Re: 'fabric1' and 'fabric2' packages
+mailing list On 2020-05-25 15:19, z...@za3k.com wrote: On 2020-05-25 12:40, Jeff Forcier wrote: - First, please understand I'm still kinda buried so I can't take any serious action on my end this week. Hopefully by June. No worries, I'm currently retired after like 3 consecutive startup jobs myself, trust me that I'm no stranger to burnout or hectic schedules. Though I have never released a popular project, that's a lot. And oof, sorry about the issue tracker stuff. Let me know if there's anything I could do? I'm planning to do everything on this but code review, just needed some big-picture decisions. I think I got all those now. Basically I am pretty good at "work on things for a week" but can lose motivation on "work on something off and on". So I like to get questions out of the way up front. This lets me binge until code review, it's perfect. - I think it's worth asking the distros if they would allow a Python 2 Fabric 1 option, for those users who are still stuck supporting Py2, but like you say I would not be surprised if they've already closed that door. Will do. I'll hand them a package and they can say yes or no (asking before making a package will get stupid responses, sadly). It's not that much work, and I'll learn what I need for a py3 fabric1 package even if they say no. - The next big thing is evaluating 'fabric3' to see where they forked / how much they added besides py3 compat. Could be anything, I've no idea. Suppose it helps that Fabric 1 dev has been nearly nil since 2 came out, but if they added a lot of features or bugfixes that's a lot for me to review. All right, I'll figure out the best way to make stuff work as you're not familiar with fabric3 either. I'll look into the current state, maybe get in touch with them if needed and they're around. - Dropping Python 2: not anytime terrifically soon, but depends how the rest of things go. The userbase for this type of tool skews conservative and supporting 2+3 isn't /that/ painful, still, it mostly just means fewer Py3-only toys. Got it, thanks. Yeah I don't think fabric will be too bad, not too much string/bytes stuff which is the big gotcha. - Pip can absolutely handle updating both major release lines at once, that's never been a problem. OK. - And I still apologize for the frustration - I was hoping v2 would get to where it was an obvious upgrade to v1, far sooner than it did. I can't have anticipated being burned out, but I should still have at least attempted some kind of "in place upgrade". Hindsight... Hindsight. I've made all these mistakes myself, trust me :).
Re: 'fabric1' and 'fabric2' packages
+mailing list, accidentally replied to Jeff/bitprophet directly On 2020-05-24 16:39, z...@za3k.com wrote: Thanks for the info, you're encouraging me. FYI my timeline is something like "this week", I don't see any big chunks of coding needed. 1. I see a couple options at this point. I'm going to ask you to tell me which one is best, because it's your project and because I don't know enough. - I write a package for fabric1 today, using python2, even though it's end-of-life, and then pursue one of the below after it's done. This is the only one where I think distro maintainers could possibly say no (because it's a new python2-based package), but I'm still willing to try it. - I start on python3 support for 1.x immediately. I can do this one, I understand all the steps. Next, pip release. Then, I make packages. - You or I talk to fabric3 folks to merge their work. Next, pip release. Then, I make packages. I don't know fabric3 licensing, are the people on board, what version of 1.x version it's forked off of. But yeah, could save testing time if it's forked off the latest 1.x and they're happy to have their changes merged. - (Longer term, after feature parity) You or I could write a 1.x API compat layer for 2.x. Pip release, no new distro package. A compat layer is a good idea I hadn't thought of. I don't know enough about 1.x vs 2.x to know how much this is just some wrappers vs how much there's some fundamental incompatibilities. This shouldn't be step 1, because feature compatibility and a compat layer are both big projects, so it would massively delay having packages. 2. I get that you wanted 1.x to keep python2 support+stability when you did the 2.x release. Want to drop python2 support for 1.x now that the official stance is that python2 is end-of-life or keep support? Will make quite a bit of difference while adding python3 support. 3. Can pip support pushing new stuff to both 1.x and 2.x, technically? 4. How's the test suite? Adding python3 support is likely to break a lot of stuff. On 2020-05-23 12:12, Jeff Forcier wrote: I'm still digging myself out of a couple large 2019/2020 related personal holes (and that was all before The Circumstances happened), but this sort of thing has been on my mind lately (blog post forthcoming). oof, life hits us all sometimes. hope you're doing okay. Thanks for reaching out about it - lots of folks seem to think that just dropping a ticket on github is the only way to communicate lately, even about major things...:'( I wouldn't take it as super meaningful. Email is losing popularity, and they may not even be aware of the list. Pin something about where to send what communications to the mailing list to your github issue tracker. Re-establish the cultural norms you want in the next generation. - though once official-fabric 3.x (and 4.x and etc) come out - ideally w/o being full rewrites and just being small chunks of API changes - that gets more difficult...I really hate running into packages named eg 'project2 version 5.8.13'. I'm assuming that the plan is for 3.x to break backwards compatibility, based on semantic versioning and the idea that one might want to maintain 3 package names. I don't think a roadmap of incrementally changing the API while breaking backwards compatibility is a good one. API stability is a pretty big deal for me as a user who uses this to provision and update all my machines. I think you want a good API, but as a user having a stable API is more important to me. Can I encourage you not to break backwards compatibility, even if it makes the codebase worse? I'd have to see the specifics to make better suggestions, but here are some generics. One common compromise approach is to have planned x-version-back support: mark some part of the API as "deprecated" for a couple versions but keep it working. (Deprecation = remove it from documentation, issue warnings at runtime, maybe other things). Then after X versions with that part of the API deprecated, actually drop support. This gives a migration path for users with no breaking steps. This plan assumes a regular release schedule, adjust appropriately if you're going to do stuff in spurts. It's equally valid to keep deprecated API functions around indefinitely, if there's no big cost, and it's great for users. Aside: The step of deprecation that developers usually suck at, is to actually tell people what specifically to do instead of using the deprecated piece. I think packaging is pretty reversible and OK to punt packaging decisions to the time of fabric 3.x, 4.x release. Probably what you'd do is have a 'fabric1' on version 1.x and a 'fabric2' on version 3.0, even if that's a little confusing. If you can come up with a better name than "fabric2" it would make things less confusing, but I can only come up with "fabric" and "fabric1" which I think is worse. I mean you could literally rename one project, but this seems like a hella dumb reason to do that. - however in OS
Re: 'fabric1' and 'fabric2' packages
+mailing list, accidentally replied to Jeff/bitprophet directly On 2020-05-25 12:40, Jeff Forcier wrote: In rough order: - First, please understand I'm still kinda buried so I can't take any serious action on my end this week. Hopefully by June. - FYI, my schedule at the dayjob is "open source Fridays" so most of my output will be that day of the week. - I think it's worth asking the distros if they would allow a Python 2 Fabric 1 option, for those users who are still stuck supporting Py2, but like you say I would not be surprised if they've already closed that door. - The next big thing is evaluating 'fabric3' to see where they forked / how much they added besides py3 compat. Could be anything, I've no idea. Suppose it helps that Fabric 1 dev has been nearly nil since 2 came out, but if they added a lot of features or bugfixes that's a lot for me to review. - Dropping Python 2: not anytime terrifically soon, but depends how the rest of things go. The userbase for this type of tool skews conservative and supporting 2+3 isn't /that/ painful, still, it mostly just means fewer Py3-only toys. - A discussion about when to drop older Py3 so we can use, e.g. f-strings or newer typing/async features, is its own thing... - Pip can absolutely handle updating both major release lines at once, that's never been a problem. - Bugs in support tools (eg my changelog software) can make it more of a pain, but those are surmountable problems. - Test suite for Fab 2 is great (should be 90+% cov), but one of the many reasons 1.x got pseudo abandoned is its test suite is ... nowhere near that good. In part because the old module-globals design precludes sane testing of all the moving parts. - This is why I resisted doing an in place rewrite or Py3 upgrade - there was a lower guarantee that the suite would catch issues. Starting over and being test-driven was incredibly compelling. - Re: compat layer: ideally it'll be possible to write something that looks like Fabric 1 but implemented on top of Fabric 2, but it'll unlikely ever be 100% so the aim should probably be to help users slowly migrate / hit the highest value API members. - Re: 3.x, 4.x etc - I didn't mean we'd put out such releases so frequently, just that they would not be the full rewrite that 2.0 was. - We're talking things like "subsystem X has gotten far too creaky and is super hard to maintain with all the shit we've added, it needs a rewrite" or "we found that the original API design for Y module is too frustrating for too many users and need to rejigger the way you call it". - As you say, the "right" way to handle a lot of this is to add-and-deprecate, but eventually you do need to cut the chaff so it's not a support burden. Feeling out the right length of time for this intermediate period is the tricky part. Off the cuff I'd look at how Django does it as they figured this out long ago. (Not that Fabric is anywhere near that size, obviously, I couldn't have even pretended to solo maintenance otherwise...) - Resentful upgrades: to be fair, I have no control over the distros, and I took pains on /my/ end to communicate and ensure that running just the old version, or running the new one side by side w/ it, would function. - But that brings us back to the classic gulf between pip users and distro-package users - such an old, frustrating story for all involved, for all libraries (except the ones who have the time to also be their own distro maintainers :D) - And I still apologize for the frustration - I was hoping v2 would get to where it was an obvious upgrade to v1, far sooner than it did. I can't have anticipated being burned out, but I should still have at least attempted some kind of "in place upgrade". Hindsight... Depending how this week goes I hope to get a blog post up on Fri or the weekend (still dealing with a ton of just-moved bullshit) expounding on my need for help from folks. Mostly in a triage sense (I basically have Github Issues PTSD now), but certainly this sort of assistance re: other aspects of maintenance that I haven't had time for, is also more than welcome. Best, Jeff On Sun, May 24, 2020 at 7:39 PM wrote: Thanks for the info, you're encouraging me. FYI my timeline is something like "this week", I don't see any big chunks of coding needed. 1. I see a couple options at this point. I'm going to ask you to tell me which one is best, because it's your project and because I don't know enough. - I write a package for fabric1 today, using python2, even though it's end-of-life, and then pursue one of the below after it's done. This is the only one where I think distro maintainers could possibly say no (because it's a new python2-based package), but I'm still willing to try it. - I start on python3 support for 1.x immediately. I can do this one, I understand all the steps. Next, pip release. Then, I make packages. - You or I talk to fabric3 folks to merge their work. Next, pip release. Then, I make packages. I don't know
Re: 'fabric1' and 'fabric2' packages
I'm still digging myself out of a couple large 2019/2020 related personal holes (and that was all before The Circumstances happened), but this sort of thing has been on my mind lately (blog post forthcoming). Thanks for reaching out about it - lots of folks seem to think that just dropping a ticket on github is the only way to communicate lately, even about major things...:'( At a very high level: - From the perspective of operating system package managers, I have no /strong/ opinions about naming re: fabric vs fabric1, fabric2, etc, but suspect you're on the right track. - because it's easy in pip to say "give me package named X, w/ version specifier Y" I decided to push Fabric 2.x releases to the 'fabric' name (as well as 'fabric2' to allow users to have both installed while migrating) - there was no real need for a 'fabric1' on pypi. - however in OS packaging land that's very NOT true, so I don't have a big problem with attempting to deprecate 'fabric' in favor of two explicit 'fabric1'/'fabric2' versions for now (however this kind of decision is often more up to the distro's packagers/policymakers than us upstream authors anyway) - though once official-fabric 3.x (and 4.x and etc) come out - ideally w/o being full rewrites and just being small chunks of API changes - that gets more difficult...I really hate running into packages named eg 'project2 version 5.8.13'. - speaking of: fabric3 on pypi is non-official and makes me sad, but mostly in a guilty fashion, which also brings me to... - I've been contemplating bringing the fabric3 folks into the fold to put out official Py3 compat 1.x releases. - I'd still prefer to get Fabric 2 to feature parity (+ perhaps even a compat layer), but things did not go as planned and I don't begrudge people wanting to use the 1.x design on Python 3. - A BIG disclaimer here is that it's conditional on breadth of divergence; "fabric 1 + python3 compat" is one thing, "a big ol' fork with a lot of additional changes" is another. I don't know which it is right now, but the whole point of 2.x and not doing py3 on 1.x was for stability's sake. - (If any of y'all are on here, please holler; I would otherwise go look up the pypi/github entries and try emailing folks myself, when I've time.) - I am not personally aware of ongoing distro-level packaging work like what you (za3k) are discussing, but that's another area where I /wish/ I had the time/energy to stay on top of things. (Again ... stay tuned for a blog post) - Others on the list (it's quiet but I assume still has folks subbed!) may have closer distro ties - hopefully they will chime in. Thanks, Jeff On Sat, May 23, 2020 at 2:01 AM za3k--- via wrote: > Hmm okay, so Python2 end-of-life was Jan 1. So actually, I think I will > offer to backport python3 support to Fabric 1.x. This one I'm going to > wait on a 'yes' before starting. > > -- Jeff Forcier Unix sysadmin; Python engineer http://bitprophet.org
Re: 'fabric1' and 'fabric2' packages
Hmm okay, so Python2 end-of-life was Jan 1. So actually, I think I will offer to backport python3 support to Fabric 1.x. This one I'm going to wait on a 'yes' before starting.
