Re: [Rd] survival changes
Terry Let me call this things to think about, rather than advice. I went through a similar process twice, once about 30 years ago and once about 20 years ago. I had fewer dependent packages of course, but still enough to cause headaches. I don't recommend doing it often. - I think you need to consider where you would like to end up before deciding how to get there. If you end up having to maintain a lot of legacy stuff I don't think you will be very happy. So then the problem becomes how to help people get off the part you want to abandon, rather then how to help them stay on it. - I know you are very experienced, but I will be really impressed if you get the new approach perfect on the first shot. That argues for having a new package with hardly any users so you can fiddle with the API more easily, and not deprecating the old one until you are really happy with the new one. - There may be a part which is common to both old and new and/or there may be a part which is what most dependent packages use. If you can separate that out as something like survivalBase it would make your life easier. That will be especial true if that part is more stable, so don't put in anything you are experimenting with. Good luck, Paul Gilbert On 6/1/19 8:02 PM, Therneau, Terry M., Ph.D. via R-devel wrote: On 6/1/19 1:32 PM, Marc Schwartz wrote: On Jun 1, 2019, at 12:59 PM, Peter Langfelder wrote: On Sat, Jun 1, 2019 at 3:22 AM Therneau, Terry M., Ph.D. via R-devel wrote: In the next version of the survival package I intend to make a non-upwardly compatable change to the survfit object. With over 600 dependent packages this is not something to take lightly, and I am currently undecided about the best way to go about it. I'm looking for advice. The change: 20+ years ago I had decided not to include the initial x=0,y=1 data point in the survfit object itself. It was not formally an estimand and the plot/points/lines etc routines could add this on themselves. That turns out to have been a mistake, and has led to a steady proliferation of extra bits as I realized that the time axis doesn't always start at 0, and later (with multi state) that y does not always start at 1 (though the states sum to 1), and later the the error doesn't always start at 0, and another realization with cumulative hazard, and ... The new survfit method for multi-state coxph models was going to add yet another special case. Basically every component is turning into a duplicate of "row 1" vs "all the others". (And inconsistently named.) Three possible solutions 1. Current working draft of survival_3.0.3: Add a 'version' element to the survfit object and a 'survfit2.3' function that converts old to new. All my downstream functions (print, plot,...) start with an "if (old) update to new" line. This has allowed me to stage updates to the functions that create survfit objects -- I expect it to happen slowly. There will also be a survfit3.2 function to go backwards. Both the forward and backwards functions leave objects alone if they are currently in the desired format. 2. Make a new class "survfit3" and the necessary 'as' functions. The package would contain plot.survfit and plot.survfit3 methods, the former a two line "convert and call the second" function. 3. Something I haven't thought of. A more "clean break" solution would be to start a whole new package (call it survival2) that would make these changes, and deprecate the current survival. You could add warnings about deprecation and urging users to switch in existing survival functions. You could continue bugfixes for survival but only add new features to survival2. The new survival2 and the current survival could live side by side on CRAN for quite some time, giving maintainers of dependent packages (and just plain users) enough time to switch. This could allow you to change/clean up other parts of the package that you could perhaps also use a rethink/rewrite, without too much concern for backward compatibility. Peter Hi, I would be cautious in going in that direction, bearing in mind that survival is a Recommended package, therefore included in the default R distribution from the R Foundation and other parties. To have two versions can/will result in substantial confusion, and I would argue against that approach. There is language in the CRAN submission policy that covers API changes, which strictly speaking, may or may not be the case here, depending upon which direction Terry elects to go: "If an update will change the package’s API and hence affect packages depending on it, it is expected that you will contact the maintainers of affected packages and suggest changes, and give them time (at least 2 weeks, ideally more) to prepare updates before submitting your updated package. Do mention in the submission email which packages are affected and that their maintainers have been informed. In order to derive the
Re: [Rd] survival changes
> In the next version of the survival package I intend to make a non-upwardly compatable > change to the survfit object. With over 600 dependent packages this is not something to > take lightly, and I am currently undecided about the best way to go about it. I'm looking > for advice. > > The change: 20+ years ago I had decided not to include the initial x=0,y=1 data point in > the survfit object itself. New Package -> Bad idea. Copying Python -> The worst idea... Version element -> Not sure I understand how that works, but probably a bad idea. If all you want to do, is add an initial data point, that shouldn't be an issue. However, I'm assuming that you want to make other more significant changes to your object. So, at face value, a new object class would be the best option, so number (2) from your list of options. Note there is another possibility. With a little bit of tricky-ness, you can check if your constructor is called by a function inside a package. In which case, you can check the publication date of that package (if published after your package), and then then respond accordingly. Then you can ask the maintainers of the other packages to update their packages, but at their own time. Abs [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] survival changes
> On Jun 1, 2019, at 12:59 PM, Peter Langfelder > wrote: > > On Sat, Jun 1, 2019 at 3:22 AM Therneau, Terry M., Ph.D. via R-devel > wrote: >> >> In the next version of the survival package I intend to make a non-upwardly >> compatable >> change to the survfit object. With over 600 dependent packages this is not >> something to >> take lightly, and I am currently undecided about the best way to go about >> it. I'm looking >> for advice. >> >> The change: 20+ years ago I had decided not to include the initial x=0,y=1 >> data point in >> the survfit object itself. It was not formally an estimand and the >> plot/points/lines etc >> routines could add this on themselves. That turns out to have been a >> mistake, and has led >> to a steady proliferation of extra bits as I realized that the time axis >> doesn't always >> start at 0, and later (with multi state) that y does not always start at 1 >> (though the >> states sum to 1), and later the the error doesn't always start at 0, and >> another >> realization with cumulative hazard, and ... >> The new survfit method for multi-state coxph models was going to add yet >> another special >> case. Basically every component is turning into a duplicate of "row 1" vs >> "all the >> others". (And inconsistently named.) >> >> Three possible solutions >> 1. Current working draft of survival_3.0.3: Add a 'version' element to the >> survfit object >> and a 'survfit2.3' function that converts old to new. All my downstream >> functions (print, >> plot,...) start with an "if (old) update to new" line. This has allowed me >> to stage >> updates to the functions that create survfit objects -- I expect it to >> happen slowly. >> There will also be a survfit3.2 function to go backwards. Both the forward >> and backwards >> functions leave objects alone if they are currently in the desired format. >> >> 2. Make a new class "survfit3" and the necessary 'as' functions. The package >> would contain >> plot.survfit and plot.survfit3 methods, the former a two line "convert and >> call the >> second" function. >> >> 3. Something I haven't thought of. > > A more "clean break" solution would be to start a whole new package > (call it survival2) that would make these changes, and deprecate the > current survival. You could add warnings about deprecation and urging > users to switch in existing survival functions. You could continue > bugfixes for survival but only add new features to survival2. The new > survival2 and the current survival could live side by side on CRAN for > quite some time, giving maintainers of dependent packages (and just > plain users) enough time to switch. This could allow you to > change/clean up other parts of the package that you could perhaps also > use a rethink/rewrite, without too much concern for backward > compatibility. > > Peter Hi, I would be cautious in going in that direction, bearing in mind that survival is a Recommended package, therefore included in the default R distribution from the R Foundation and other parties. To have two versions can/will result in substantial confusion, and I would argue against that approach. There is language in the CRAN submission policy that covers API changes, which strictly speaking, may or may not be the case here, depending upon which direction Terry elects to go: "If an update will change the package’s API and hence affect packages depending on it, it is expected that you will contact the maintainers of affected packages and suggest changes, and give them time (at least 2 weeks, ideally more) to prepare updates before submitting your updated package. Do mention in the submission email which packages are affected and that their maintainers have been informed. In order to derive the reverse dependencies of a package including the addresses of maintainers who have to be notified upon changes, the function reverse_dependencies_with_maintainers is available from the developer website." Given the potential extent and impact of the changes being considered, it would seem reasonable to: 1. Post a note to R-Devel (possibly R-Help to cover a larger useR base) regarding whatever changes are finalized and formally announce them. The changes are likely to affect end useRs as well as package maintainers. 2. Send communications directly via e-mail to the relevant package maintainers that have dependencies on survival. 3. Consider a longer deprecation time frame for relevant functions, to raise awareness and allow for changes to be made by package maintainers and useRs as may be apropos. Perhaps post reminders to R-Help at relevant time points in advance as you approach the formal deprecation and release of the updated package. Terry, if you have not used it yet and/or are not aware of it, take a look at ?Deprecated in base: https://stat.ethz.ch/R-manual/R-devel/library/base/html/Deprecated.html which is helpful in setting up a deprecation process. If you
Re: [Rd] survival changes
On Sat, Jun 1, 2019 at 3:22 AM Therneau, Terry M., Ph.D. via R-devel wrote: > > In the next version of the survival package I intend to make a non-upwardly > compatable > change to the survfit object. With over 600 dependent packages this is not > something to > take lightly, and I am currently undecided about the best way to go about it. > I'm looking > for advice. > > The change: 20+ years ago I had decided not to include the initial x=0,y=1 > data point in > the survfit object itself. It was not formally an estimand and the > plot/points/lines etc > routines could add this on themselves. That turns out to have been a > mistake, and has led > to a steady proliferation of extra bits as I realized that the time axis > doesn't always > start at 0, and later (with multi state) that y does not always start at 1 > (though the > states sum to 1), and later the the error doesn't always start at 0, and > another > realization with cumulative hazard, and ... > The new survfit method for multi-state coxph models was going to add yet > another special > case. Basically every component is turning into a duplicate of "row 1" vs > "all the > others". (And inconsistently named.) > > Three possible solutions > 1. Current working draft of survival_3.0.3: Add a 'version' element to the > survfit object > and a 'survfit2.3' function that converts old to new. All my downstream > functions (print, > plot,...) start with an "if (old) update to new" line. This has allowed me > to stage > updates to the functions that create survfit objects -- I expect it to happen > slowly. > There will also be a survfit3.2 function to go backwards. Both the forward > and backwards > functions leave objects alone if they are currently in the desired format. > > 2. Make a new class "survfit3" and the necessary 'as' functions. The package > would contain > plot.survfit and plot.survfit3 methods, the former a two line "convert and > call the > second" function. > > 3. Something I haven't thought of. A more "clean break" solution would be to start a whole new package (call it survival2) that would make these changes, and deprecate the current survival. You could add warnings about deprecation and urging users to switch in existing survival functions. You could continue bugfixes for survival but only add new features to survival2. The new survival2 and the current survival could live side by side on CRAN for quite some time, giving maintainers of dependent packages (and just plain users) enough time to switch. This could allow you to change/clean up other parts of the package that you could perhaps also use a rethink/rewrite, without too much concern for backward compatibility. Peter __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
Re: [Rd] survival changes
On Sat, Jun 1, 2019 at 5:22 AM Therneau, Terry M., Ph.D. via R-devel wrote: > > In the next version of the survival package I intend to make a non-upwardly > compatable > change to the survfit object. With over 600 dependent packages this is not > something to > take lightly, and I am currently undecided about the best way to go about it. > I'm looking > for advice. > I encountered several issues like this while making a change to xts. I encountered several buggy and inconsistent behaviors while moving attributes from the xts object to the xts object's index attribute. I don't want to "fix" some of these in the next version, because downstream packages may rely on the old behavior. Though it's specific to Python, "API Evolution the Right Way" has many good strategies. https://emptysqua.re/blog/api-evolution-the-right-way/ The "Deleting Features" and "Changing Behavior" sections are most relevant to your situation. > The change: 20+ years ago I had decided not to include the initial x=0,y=1 > data point in > the survfit object itself. It was not formally an estimand and the > plot/points/lines etc > routines could add this on themselves. That turns out to have been a > mistake, and has led > to a steady proliferation of extra bits as I realized that the time axis > doesn't always > start at 0, and later (with multi state) that y does not always start at 1 > (though the > states sum to 1), and later the the error doesn't always start at 0, and > another > realization with cumulative hazard, and ... > The new survfit method for multi-state coxph models was going to add yet > another special > case. Basically every component is turning into a duplicate of "row 1" vs > "all the > others". (And inconsistently named.) > > Three possible solutions > 1. Current working draft of survival_3.0.3: Add a 'version' element to the > survfit object > and a 'survfit2.3' function that converts old to new. All my downstream > functions (print, > plot,...) start with an "if (old) update to new" line. This has allowed me > to stage > updates to the functions that create survfit objects -- I expect it to happen > slowly. > There will also be a survfit3.2 function to go backwards. Both the forward > and backwards > functions leave objects alone if they are currently in the desired format. > This seems reasonable. It would also give you opportunity to warn users if functions that expect a new object receive an old object. That would help them convert any of their functions that rely on the old structure/behavior. You could also add a global option to enable warnings with the default set to FALSE for the first release. Let users know they can set that option to TRUE to identify places where they may need to modify their use cases. > 2. Make a new class "survfit3" and the necessary 'as' functions. The package > would contain > plot.survfit and plot.survfit3 methods, the former a two line "convert and > call the > second" function. > Rather than an entirely new class, I wonder if you could instead make the new class a subclass. I'm not sure whether this has advantages over adding an element, but it's another possibility. > 3. Something I haven't thought of. > I don't have any other ideas, but I would be happy to discuss offline if that would be helpful. > Number 2 has a cleanness about it, but there is a long term nuisance about it > wrt > documentation. Users, not unreasonably, expect the survfit function to > produce a survfit > object, and that is what they look for in the help pages. > > I plan to have 3.0-x on github before userR so that users can begin to play > with it (and > to get feeback before pushing to CRAN), so need to make a decision. > > Terry T. > > > > [[alternative HTML version deleted]] > > __ > R-devel@r-project.org mailing list > https://stat.ethz.ch/mailman/listinfo/r-devel -- Joshua Ulrich | about.me/joshuaulrich FOSS Trading | www.fosstrading.com R/Finance 2019 | www.rinfinance.com __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
[Rd] survival changes
In the next version of the survival package I intend to make a non-upwardly compatable change to the survfit object. With over 600 dependent packages this is not something to take lightly, and I am currently undecided about the best way to go about it. I'm looking for advice. The change: 20+ years ago I had decided not to include the initial x=0,y=1 data point in the survfit object itself. It was not formally an estimand and the plot/points/lines etc routines could add this on themselves. That turns out to have been a mistake, and has led to a steady proliferation of extra bits as I realized that the time axis doesn't always start at 0, and later (with multi state) that y does not always start at 1 (though the states sum to 1), and later the the error doesn't always start at 0, and another realization with cumulative hazard, and ... The new survfit method for multi-state coxph models was going to add yet another special case. Basically every component is turning into a duplicate of "row 1" vs "all the others". (And inconsistently named.) Three possible solutions 1. Current working draft of survival_3.0.3: Add a 'version' element to the survfit object and a 'survfit2.3' function that converts old to new. All my downstream functions (print, plot,...) start with an "if (old) update to new" line. This has allowed me to stage updates to the functions that create survfit objects -- I expect it to happen slowly. There will also be a survfit3.2 function to go backwards. Both the forward and backwards functions leave objects alone if they are currently in the desired format. 2. Make a new class "survfit3" and the necessary 'as' functions. The package would contain plot.survfit and plot.survfit3 methods, the former a two line "convert and call the second" function. 3. Something I haven't thought of. Number 2 has a cleanness about it, but there is a long term nuisance about it wrt documentation. Users, not unreasonably, expect the survfit function to produce a survfit object, and that is what they look for in the help pages. I plan to have 3.0-x on github before userR so that users can begin to play with it (and to get feeback before pushing to CRAN), so need to make a decision. Terry T. [[alternative HTML version deleted]] __ R-devel@r-project.org mailing list https://stat.ethz.ch/mailman/listinfo/r-devel
