Re: [covid-19] Reviving tensorflow packaging effort (Was: Missing dependancies for streamlit)
On Mon, Apr 27, 2020 at 02:05:58AM +, Mo Zhou wrote: > On Sat, Apr 25, 2020 at 01:42:28PM +0200, Andreas Tille wrote: > > > Is there any COVID-19 package using pytorch blocked due to its absense? > > > > I admit I can not say without doing detailed research on the set of > > relevant packages[3] > > I quickly went through all these projects and none of them seems to > depend on pytorch. So I think I'd better treat these packages as normal > ones instead of occupying the resources. OK, may be not our list - but this list is known to be incomplete. I quickly found at least one link that is connecting pytorch with COVID-19: https://github.com/IliasPap/COVIDNet In case it might convince you that pytorch is by no means off topic I would add this project. > > > https://salsa.debian.org/deeplearning-team > > > > Cool. I've added this to the Blends machine-readable gatherer since > > several interesting Blends packages are there. It would be great if > > you could add these to the according Debian Science and Debian Med > > tasks. > > I've granted Debian Science Team and Debian Med Team with "Maintainer" > access to Debian Deep Learning Team. Sounds very sensible. > I reused the mail address of debian science team as the maintainer > mail address of deep learning team. So people won't have to subscribe > to a @tracker mail address. Thank you for this. > > BTW, what might be interesting for you: Olek is very actively working > > on bazel: > > > >https://salsa.debian.org/olek/bazel > > That's great. Definitely. Kind regards Andreas. -- http://fam-tille.de
Re: [covid-19] Reviving tensorflow packaging effort (Was: Missing dependancies for streamlit)
On Sat, Apr 25, 2020 at 01:42:28PM +0200, Andreas Tille wrote: > > Is there any COVID-19 package using pytorch blocked due to its absense? > > I admit I can not say without doing detailed research on the set of > relevant packages[3] I quickly went through all these projects and none of them seems to depend on pytorch. So I think I'd better treat these packages as normal ones instead of occupying the resources. > > https://salsa.debian.org/deeplearning-team > > Cool. I've added this to the Blends machine-readable gatherer since > several interesting Blends packages are there. It would be great if > you could add these to the according Debian Science and Debian Med > tasks. I've granted Debian Science Team and Debian Med Team with "Maintainer" access to Debian Deep Learning Team. I reused the mail address of debian science team as the maintainer mail address of deep learning team. So people won't have to subscribe to a @tracker mail address. > BTW, what might be interesting for you: Olek is very actively working > on bazel: > >https://salsa.debian.org/olek/bazel That's great.
Re: [covid-19] Reviving tensorflow packaging effort (Was: Missing dependancies for streamlit)
On Sat, Apr 25, 2020, 12:36 Mo Zhou wrote: > Hi Tille, > > Is there any COVID-19 package using pytorch blocked due to its absense? > > A good news is that I've managed to strip the whole third_party/ > directory of src:pytorch, and started to forward my patches to upstream[1]. > When all my modifications entered the upstream repo, I'll be quite > confident that our src:pytorch package can enter the archive without > any (annoying) embedded sources [2]. > > What I'm doing now is to wait for the upstream to merge my commits, and > for the ftp-masters to accept my NEW dependency packages. > > In that sense, I'd like to take the COVID-19 shortcut to pass NEW > quickly, if any COVID-19 related package needs pytorch. > > --- According to my status page > https://qa.debian.org/developer.php?login=lumin > these are the NEW dependency packages: > > fp16 > fxdiv > gloo > onnx > psimd > pthreadpool > Go ahead and add them to https://salsa.debian.org/med-team/community/2020-covid19-hackathon/-/wikis/NEW-Requests > I'll start to re-debianize src:pytorch from scratch when all of my > commits had been upstreamed. > Any reason to not add the removed files/folders to the Files-Excluded stanza of debian/copyright and repack the source while you wait? > --- all of my on-going work are publically available: > https://salsa.debian.org/deeplearning-team > https://github.com/cdluminate/pytorch (messy, not rebased yet) > The links to my PRs can be found on [1] > > [1] https://github.com/pytorch/pytorch/issues/14699 > [2] Finally, we will have a modern deep learning framework in the > archive. It's better than having nothing even if I'm working on the > cpu-only (free) version. > > On Wed, Apr 08, 2020 at 10:06:12AM +0200, Andreas Tille wrote: > > Hi Mo, > > > > On Wed, Apr 08, 2020 at 02:46:06AM +, Mo Zhou wrote: > > > On Tue, Apr 07, 2020 at 11:49:07AM +0200, Andreas Tille wrote: > > > > On Tue, Apr 07, 2020 at 08:56:45AM +0100, Rebecca N. Palmer wrote: > > > > > tensorflow 1.10 was packaged in experimental, but with reduced > performance, > > > > > and was removed because this was considered not worth it: > > > > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=935769 > > > > > > > > Ah I simply forgot this. Thanks for refreshing my mind. > > > > > > After that I tried to refresh the packaging and uploaded tensorflow 2.0 > > > to the NEW queue. The ftp-master complained about the embedded snapshot > > > version of Eigen3. Ftp-masters are not convinced even if I said > > > tensorflow FTBFS against the version shipped in our archive, and it > > > could waste lots of my time and energy to patch the related code. > > > > I can understand your feeling. I've re-read the discussion[1] about > > including eigen3 into the tensorflow source. The most promising > > statement was given in > > > > > https://alioth-lists.debian.net/pipermail/debian-science-maintainers/2020-March/079169.html > > > >Sean Whitton spwhitton at spwhitton.name > >Tue Mar 3 04:04:52 GMT 2020 > > > >Rejected per your request, but from my point of view this discussion > is > >not over -- Policy 4.13 says that packages *should* not use > convenience > >copies of code, not that they *must* not. > > > >Thank you for all your work on uploading useful ML packages. > > > > At first: Thanks also from me! > > > > >From my point of view that is not a lost case so we should really try > > again. > > > > > I was so angry at that time so I deleted my name from the maintainers > > > and wrote the git message "I'm dropping this burden.". > > > > Well, sometimes personal feelings are dominating our actions. I hope we > > could form some real team around this to spread the technical as well as > > the organisational burden. > > > > > So another kind notice for whoever is willing to take over tensorflow: > > > also be prepared to fuss with ftp-master. > > > > I need to admit that I'm absolutely happy about ftpmaster. They are > > currently *extremely* supportive to our COVID-19 hackathon and > 30 > > packages made it from upload to unstable in less than 24 hours! > > > > Since I consider it a "promising time" to reach a lot for deep learning > > tools in Debian which are frequently used in those tools to hunt down > > COVID-19 I'd like to call for help here for trying again to get > > tensorflow in - this time even with the Python3 module. > > > > Lumin has written an own build system for the C++ library since he has > > found out that the upstream build system is not usuable for Debian. > > Regarding the Python3 module he said that its not simply a matter of > > adapting his build system but "significantly extend" it since the python > > building process is much more complicated than the process for C++. > > > > Any takers for this task? > > > > Kind regards > > > >Andreas. > > > > > > [1] > https://alioth-lists.debian.net/pipermail/debian-science-maintainers/2020-March/079054.html > > > > -- > > http://fam-tille.de > > >
Re: [covid-19] Reviving tensorflow packaging effort (Was: Missing dependancies for streamlit)
Hi Mo, On Sat, Apr 25, 2020 at 10:20:32AM +, Mo Zhou wrote: > > Is there any COVID-19 package using pytorch blocked due to its absense? I admit I can not say without doing detailed research on the set of relevant packages[3] > A good news is that I've managed to strip the whole third_party/ > directory of src:pytorch, and started to forward my patches to upstream[1]. > When all my modifications entered the upstream repo, I'll be quite > confident that our src:pytorch package can enter the archive without > any (annoying) embedded sources [2]. Cool! > What I'm doing now is to wait for the upstream to merge my commits, and > for the ftp-masters to accept my NEW dependency packages. > > In that sense, I'd like to take the COVID-19 shortcut to pass NEW > quickly, if any COVID-19 related package needs pytorch. Frankly speaking: A lot of the relevant high level tools of epidemiology are using deep learning technologies. So its not really wrong to say its relevant for COVID-19 fighting. > --- According to my status page > https://qa.debian.org/developer.php?login=lumin > these are the NEW dependency packages: > > fp16 > fxdiv > gloo > onnx > psimd > pthreadpool > > I'll start to re-debianize src:pytorch from scratch when all of my > commits had been upstreamed. > > --- all of my on-going work are publically available: > https://salsa.debian.org/deeplearning-team Cool. I've added this to the Blends machine-readable gatherer since several interesting Blends packages are there. It would be great if you could add these to the according Debian Science and Debian Med tasks. BTW, what might be interesting for you: Olek is very actively working on bazel: https://salsa.debian.org/olek/bazel > https://github.com/cdluminate/pytorch (messy, not rebased yet) > The links to my PRs can be found on [1] > > [1] https://github.com/pytorch/pytorch/issues/14699 > [2] Finally, we will have a modern deep learning framework in the > archive. It's better than having nothing even if I'm working on the > cpu-only (free) version. Thanks a lot for all your work. Its extremely helpful. Kind regards Andreas. [3] https://salsa.debian.org/med-team/community/2020-covid19-hackathon/-/wikis/COVID-19-Hackathon-packages-needing-work -- http://fam-tille.de
Re: [covid-19] Reviving tensorflow packaging effort (Was: Missing dependancies for streamlit)
Hi Tille, Is there any COVID-19 package using pytorch blocked due to its absense? A good news is that I've managed to strip the whole third_party/ directory of src:pytorch, and started to forward my patches to upstream[1]. When all my modifications entered the upstream repo, I'll be quite confident that our src:pytorch package can enter the archive without any (annoying) embedded sources [2]. What I'm doing now is to wait for the upstream to merge my commits, and for the ftp-masters to accept my NEW dependency packages. In that sense, I'd like to take the COVID-19 shortcut to pass NEW quickly, if any COVID-19 related package needs pytorch. --- According to my status page https://qa.debian.org/developer.php?login=lumin these are the NEW dependency packages: fp16 fxdiv gloo onnx psimd pthreadpool I'll start to re-debianize src:pytorch from scratch when all of my commits had been upstreamed. --- all of my on-going work are publically available: https://salsa.debian.org/deeplearning-team https://github.com/cdluminate/pytorch (messy, not rebased yet) The links to my PRs can be found on [1] [1] https://github.com/pytorch/pytorch/issues/14699 [2] Finally, we will have a modern deep learning framework in the archive. It's better than having nothing even if I'm working on the cpu-only (free) version. On Wed, Apr 08, 2020 at 10:06:12AM +0200, Andreas Tille wrote: > Hi Mo, > > On Wed, Apr 08, 2020 at 02:46:06AM +, Mo Zhou wrote: > > On Tue, Apr 07, 2020 at 11:49:07AM +0200, Andreas Tille wrote: > > > On Tue, Apr 07, 2020 at 08:56:45AM +0100, Rebecca N. Palmer wrote: > > > > tensorflow 1.10 was packaged in experimental, but with reduced > > > > performance, > > > > and was removed because this was considered not worth it: > > > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=935769 > > > > > > Ah I simply forgot this. Thanks for refreshing my mind. > > > > After that I tried to refresh the packaging and uploaded tensorflow 2.0 > > to the NEW queue. The ftp-master complained about the embedded snapshot > > version of Eigen3. Ftp-masters are not convinced even if I said > > tensorflow FTBFS against the version shipped in our archive, and it > > could waste lots of my time and energy to patch the related code. > > I can understand your feeling. I've re-read the discussion[1] about > including eigen3 into the tensorflow source. The most promising > statement was given in > > > https://alioth-lists.debian.net/pipermail/debian-science-maintainers/2020-March/079169.html > >Sean Whitton spwhitton at spwhitton.name >Tue Mar 3 04:04:52 GMT 2020 > >Rejected per your request, but from my point of view this discussion is >not over -- Policy 4.13 says that packages *should* not use convenience >copies of code, not that they *must* not. > >Thank you for all your work on uploading useful ML packages. > > At first: Thanks also from me! > > >From my point of view that is not a lost case so we should really try > again. > > > I was so angry at that time so I deleted my name from the maintainers > > and wrote the git message "I'm dropping this burden.". > > Well, sometimes personal feelings are dominating our actions. I hope we > could form some real team around this to spread the technical as well as > the organisational burden. > > > So another kind notice for whoever is willing to take over tensorflow: > > also be prepared to fuss with ftp-master. > > I need to admit that I'm absolutely happy about ftpmaster. They are > currently *extremely* supportive to our COVID-19 hackathon and > 30 > packages made it from upload to unstable in less than 24 hours! > > Since I consider it a "promising time" to reach a lot for deep learning > tools in Debian which are frequently used in those tools to hunt down > COVID-19 I'd like to call for help here for trying again to get > tensorflow in - this time even with the Python3 module. > > Lumin has written an own build system for the C++ library since he has > found out that the upstream build system is not usuable for Debian. > Regarding the Python3 module he said that its not simply a matter of > adapting his build system but "significantly extend" it since the python > building process is much more complicated than the process for C++. > > Any takers for this task? > > Kind regards > >Andreas. > > > [1] > https://alioth-lists.debian.net/pipermail/debian-science-maintainers/2020-March/079054.html > > -- > http://fam-tille.de >
[covid-19] Reviving tensorflow packaging effort (Was: Missing dependancies for streamlit)
Hi Mo, On Wed, Apr 08, 2020 at 02:46:06AM +, Mo Zhou wrote: > On Tue, Apr 07, 2020 at 11:49:07AM +0200, Andreas Tille wrote: > > On Tue, Apr 07, 2020 at 08:56:45AM +0100, Rebecca N. Palmer wrote: > > > tensorflow 1.10 was packaged in experimental, but with reduced > > > performance, > > > and was removed because this was considered not worth it: > > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=935769 > > > > Ah I simply forgot this. Thanks for refreshing my mind. > > After that I tried to refresh the packaging and uploaded tensorflow 2.0 > to the NEW queue. The ftp-master complained about the embedded snapshot > version of Eigen3. Ftp-masters are not convinced even if I said > tensorflow FTBFS against the version shipped in our archive, and it > could waste lots of my time and energy to patch the related code. I can understand your feeling. I've re-read the discussion[1] about including eigen3 into the tensorflow source. The most promising statement was given in https://alioth-lists.debian.net/pipermail/debian-science-maintainers/2020-March/079169.html Sean Whitton spwhitton at spwhitton.name Tue Mar 3 04:04:52 GMT 2020 Rejected per your request, but from my point of view this discussion is not over -- Policy 4.13 says that packages *should* not use convenience copies of code, not that they *must* not. Thank you for all your work on uploading useful ML packages. At first: Thanks also from me! >From my point of view that is not a lost case so we should really try again. > I was so angry at that time so I deleted my name from the maintainers > and wrote the git message "I'm dropping this burden.". Well, sometimes personal feelings are dominating our actions. I hope we could form some real team around this to spread the technical as well as the organisational burden. > So another kind notice for whoever is willing to take over tensorflow: > also be prepared to fuss with ftp-master. I need to admit that I'm absolutely happy about ftpmaster. They are currently *extremely* supportive to our COVID-19 hackathon and > 30 packages made it from upload to unstable in less than 24 hours! Since I consider it a "promising time" to reach a lot for deep learning tools in Debian which are frequently used in those tools to hunt down COVID-19 I'd like to call for help here for trying again to get tensorflow in - this time even with the Python3 module. Lumin has written an own build system for the C++ library since he has found out that the upstream build system is not usuable for Debian. Regarding the Python3 module he said that its not simply a matter of adapting his build system but "significantly extend" it since the python building process is much more complicated than the process for C++. Any takers for this task? Kind regards Andreas. [1] https://alioth-lists.debian.net/pipermail/debian-science-maintainers/2020-March/079054.html -- http://fam-tille.de
