As I said - depending on the precondition for how tightly petsc & petsc4py should be synced - we can choose the appropriate model.
[for the metioned precondition - I was suggestion the model]. As you say - the initilal phrasing is wrong - so the model is not suitable. A single repo does make it easy for 'all feature branches' to be in sync.. In that model with-petsc4py=1 would be the default. However anyone changing the petsc UI would also have to update petsc4py with the change.. [and Lisandro says - that change could be buggy - and he would like to review it -before its applied] For 2 repos model - perhaps sync at 'master' might work. Satish On Thu, 22 Oct 2015, Barry Smith wrote: > > > On Oct 22, 2015, at 9:53 PM, Satish Balay <[email protected]> wrote: > > > > If dmplex with petsc4py is the primary use case [and assuming the > > usage is primarily --download-petsc4py] - we chould use the first > > model I mentioned - and that should not be too difficult. > > > > The workflow would be something like: > > > > - Matt would do the updates in petsc/dmplex feature branch & petsc4py > > dmplex branches simultaneosusly. > > That's nuts; Matt is a college professor he doesn't have time to keep two > branches in sync for no good reason when they could be in the same repository. > > Plus other people will be making other changes in PETSc that break petsc4py > > This discussion has shown to me that keeping petsc and petsc4py in > different repositories is just silly and unproductive. > > > > - And would always use --download-petsc4py for all his build tests > > [we'll have to fix auto-update for --download-pkg for git version of > > pkg] > > > > - Any isues that come up - Matt will fix in petsc4py/dmplex and update > > petsc4py [before this branch is ever merged to next] > > > > So after next testing - a merge to master - Matt could issue a pull > > requst to Lisandro [or leave it for Lisandro to figureout when > > petsc4py-master gets updated with the fix]. > > > > The update would involve merging/updating petsc4py-master [and and > > update to petsc4py.py in petsc-master] > > > > [even if this petsc4py-master update doesn't happen immediately] it > > won't affect petsc4py/dmplex users as they would be using > > --download-petsc4py [which will use the working petsc4py.commitid set > > my Matt anyway] > > > > If the usage is not via --download-petsc4py - then there could be a > > more appropriate workflow for that.. [and a different testsuite to > > match that usage..] > > > > Satish > > > > On Thu, 22 Oct 2015, Barry Smith wrote: > > > >> > >> The model we adopt also depends on how cutting edge petsc4py users want > >> to be with petsc. For example if people want to use dmplex with petsc4py > >> then the two packages have to move very closely in sync. > >> > >>> On Oct 22, 2015, at 9:11 PM, Satish Balay <[email protected]> wrote: > >>> > >>> On Thu, 22 Oct 2015, Barry Smith wrote: > >>> > >>>> > >>>>> On Oct 22, 2015, at 7:28 PM, Satish Balay <[email protected]> wrote: > >>>>> > >>>>> The current infrastruture tests --download-petsc4py in petsc 'next' > >>>>> and 'master' branches. > >>>>> > >>>>> Wrt next - usually a change in a feature branch (when it gets merged > >>>>> to next) triggers this breakage. And usual thing to do is to fix it > >>>>> in the feature branch (and merge to next again) > >>>>> > >>>>> Wrt to petsc4py - this would mean updating petsc4py.py [in petsc] with > >>>>> the petsc4py repo gitcommit [optionally also tarball ] that > >>>>> corresponds to the fix. [one way to handle this on petsc4py repo is to > >>>>> create a feature branch for this fix] > >>>> > >>>> Yeah this is nuts, having to maintaining matching branches between petsc > >>>> and petsc4py and merge them at the same time into next and then master. > >>>> Clearly no one will ever do this and hence petsc4py will always be out > >>>> of step from PETSc. I do really believe that incorporating petsc4py into > >>>> PETSc would just make all these difficulties go away. For example > >>>> maintaining Tao is a breeze now since it is updated in the same branches > >>>> as PETSc. > >>>> > >>> > >>> Perhaps I should rephrase the issue in the following way: > >>> > >>> We should decide on what level petsc4py and petsc should be synced. > >>> [and the testsuite should be modified appropriately for that] > >>> > >>> Currently its done at next, master branchs. For this to work - the > >>> above - previously mentioned workflow is required. [and for this > >>> workflow - a single repo does makes thing easier] > >>> > >>> We could say sync should happen only between petsc4py-master & > >>> petsc-master. [so issues - if any in feature branches/next wont't be > >>> dealt with - and its ok for the master branches to be out of sync for > >>> a brief time for the fix to propergate]. > >>> > >>> For this - we would have a different workflow [wrt syncing petsc4py > >>> wrt petsc - and a slightly different test suite - testing only > >>> master. [We could continue having the curent test suite - but ignore > >>> errors in next]. And e-mails to Lisandro will trigger only on > >>> petsc4py errors on master. Lisandro would then fix petsc4py (master) > >>> - and then update petsc4py.py in petsc-master. > >>> > >>> Or perhaps some other mode is preferable.. > >>> > >>> Satish > >>> > >>>> Barry > >>>> > >>>>> > >>>>> When this (petsc) feature branch is merged into master - the > >>>>> corresponding feature branch on the petsc4py side can be merged into > >>>>> petsc4py master [if petsc4py-master is to be in sync with > >>>>> petsc-master]. > >>>>> > >>>>> [if there are multiple feature branches on the petsc side requiring > >>>>> corresponding petsc4py changes - somehow that has to be handled] > >>>>> > >>>>> Perhaps testing petsc4py via petsc infrastructure [using > >>>>> --download-petsc4py] is not the appropriate approach? > >>>>> > >>>>> Or we should do this only on master - not next? [or just have a > >>>>> single --download-petsc4py test - thats not mixed with other tests? - > >>>>> so its breakage is not critical for other things] > >>>>> > >>>>> Also we don't currently run any petsc4py tests. [we just test if it > >>>>> compiles or not] > >>>>> > >>>>> Satish > >>>>> > >>>>> On Thu, 22 Oct 2015, Barry Smith wrote: > >>>>> > >>>>>> > >>>>>> Lisandro, > >>>>>> > >>>>>> It is tested nightly but there is apparently currently no email sent > >>>>>> on failure. See the bottom of, for example, > >>>>>> http://ftp.mcs.anl.gov/pub/petsc/nightlylogs/archive/2015/10/22/make_next_arch-linux-pkgs-opt_crank.log > >>>>>> > >>>>>> I think what should happen is the petsc4py log file should be > >>>>>> published to the web (like the other log files) and a script look > >>>>>> through it and decide to send email if it detects a problem. Like we > >>>>>> currently have for the other log files. Someone has to dig into the > >>>>>> nightly test infrastructure to figure out where the new logic needs to > >>>>>> go. Currently Satish and Karl understand that infrastructure the best. > >>>>>> > >>>>>> Barry > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>> > >>>>>>> On Oct 22, 2015, at 6:44 PM, Lisandro Dalcin <[email protected]> > >>>>>>> wrote: > >>>>>>> > >>>>>>> On 22 October 2015 at 23:20, Barry Smith <[email protected]> wrote: > >>>>>>>> > >>>>>>>> Lisandro, > >>>>>>>> > >>>>>>>> You never responded to this. I assume it was because you did not > >>>>>>>> like the idea? This is not an attempt to take control of petsc4py or > >>>>>>>> to take credit away from petsc4py from you. Petsc4py is your package > >>>>>>>> and you will always be the one who deserves the credit. > >>>>>>>> > >>>>>>> > >>>>>>> Oh, sorry! I missed this email. > >>>>>>> > >>>>>>> > >>>>>>>> The problem is that mutual efficient development of very intertwined > >>>>>>>> packages is difficult if they are in different respositories. For > >>>>>>>> example petsc4py has been out of date with petsc master for many > >>>>>>>> weeks now because it is too much effort to update the petsc4py > >>>>>>>> repository each time we make some change in PETSc. Thus it becomes a > >>>>>>>> burden on you to go in every once in a while and fix up petsc4py. > >>>>>>>> With one repository changes would happen in both PETSc source and > >>>>>>>> petsc4py source at the same time and tests would detect problems > >>>>>>>> immediately. > >>>>>>>> > >>>>>>>> Thoughts? > >>>>>>>> > >>>>>>> > >>>>>>> I general terms, I prefer to keep things separate as much a possible. > >>>>>>> IMHO, a much important step would be to incorporate petsc4py to the > >>>>>>> nightly tests, clearly separating the petsc and petsc4py tests runs. I > >>>>>>> would love to get an email every time petsc4py nightly breaks, this > >>>>>>> way I would be able to push fixes within a day or two, say. > >>>>>>> Maintaining petsc4py requires knowledge of PETSc+Python+Cython, so > >>>>>>> ultimately I'm the one in best position to do such work, if you guys > >>>>>>> start adding commits to petsc4py withing the main petsc repo, I'm > >>>>>>> likely going to miss these changes. For example, today Michael Lange > >>>>>>> made some changes, I got an email from Bitbucket, then did a review > >>>>>>> and spotted a few issues. > >>>>>>> > >>>>>>> My hesitation to incorporate petsc4py within petsc repo is, in > >>>>>>> Python's [and linear algebra :-)] Zen : "Sparse is better than dense". > >>>>>>> If we continue adding stuff to core PETSc, at same point we will endup > >>>>>>> like Trilinos ( DON'T MISS the sencond entry in the FAQ: > >>>>>>> https://trilinos.org/download/public-git-repository/). > >>>>>>> > >>>>>>> All that being said, I don't have any problem at all adding petsc4py > >>>>>>> to the main PETSc repository. But I still think this is not a good > >>>>>>> idea, it will not solve all the problems. Nightly tests with automatic > >>>>>>> notifications to me about breakages is perhaps is all what we need to > >>>>>>> keep petsc4py in close sync with petsc/master. What about trying this > >>>>>>> first to see how it goes? > >>>>>>> > >>>>>>> > >>>>>>> PS: Please understand my position has nothing to do with credits about > >>>>>>> the project or ego-related crap. Actually, your proposal would mean > >>>>>>> less responsibility and maintenance work for me. But I don't think it > >>>>>>> is a move in the right direction. > >>>>>>> > >>>>>>> > >>>>>>> -- > >>>>>>> Lisandro Dalcin > >>>>>>> ============ > >>>>>>> Research Scientist > >>>>>>> Computer, Electrical and Mathematical Sciences & Engineering (CEMSE) > >>>>>>> Numerical Porous Media Center (NumPor) > >>>>>>> King Abdullah University of Science and Technology (KAUST) > >>>>>>> http://numpor.kaust.edu.sa/ > >>>>>>> > >>>>>>> 4700 King Abdullah University of Science and Technology > >>>>>>> al-Khawarizmi Bldg (Bldg 1), Office # 4332 > >>>>>>> Thuwal 23955-6900, Kingdom of Saudi Arabia > >>>>>>> http://www.kaust.edu.sa > >>>>>>> > >>>>>>> Office Phone: +966 12 808-0459 > >> > >> > > > >
