Re: [covid-19] Reviving tensorflow packaging effort (Was: Missing dependancies for streamlit)

[Michael Crusoe]

On Sat, Apr 25, 2020, 12:36 Mo Zhou  wrote:

> Hi Tille,
>
> Is there any COVID-19 package using pytorch blocked due to its absense?
>
> A good news is that I've managed to strip the whole third_party/
> directory of src:pytorch, and started to forward my patches to upstream[1].
> When all my modifications entered the upstream repo, I'll be quite
> confident that our src:pytorch package can enter the archive without
> any (annoying) embedded sources [2].
>
> What I'm doing now is to wait for the upstream to merge my commits, and
> for the ftp-masters to accept my NEW dependency packages.
>
> In that sense, I'd like to take the COVID-19 shortcut to pass NEW
> quickly, if any COVID-19 related package needs pytorch.
>
> --- According to my status page
> https://qa.debian.org/developer.php?login=lumin
> these are the NEW dependency packages:
>
> fp16
> fxdiv
> gloo
> onnx
> psimd
> pthreadpool
>

Go ahead and add them to
https://salsa.debian.org/med-team/community/2020-covid19-hackathon/-/wikis/NEW-Requests



> I'll start to re-debianize src:pytorch from scratch when all of my
> commits had been upstreamed.
>

Any reason to not add the removed files/folders to the Files-Excluded
stanza of debian/copyright and repack the source while you wait?


> --- all of my on-going work are publically available:
> https://salsa.debian.org/deeplearning-team
> https://github.com/cdluminate/pytorch (messy, not rebased yet)
> The links to my PRs can be found on [1]
>
> [1] https://github.com/pytorch/pytorch/issues/14699
> [2] Finally, we will have a modern deep learning framework in the
> archive. It's better than having nothing even if I'm working on the
> cpu-only (free) version.
>
> On Wed, Apr 08, 2020 at 10:06:12AM +0200, Andreas Tille wrote:
> > Hi Mo,
> >
> > On Wed, Apr 08, 2020 at 02:46:06AM +, Mo Zhou wrote:
> > > On Tue, Apr 07, 2020 at 11:49:07AM +0200, Andreas Tille wrote:
> > > > On Tue, Apr 07, 2020 at 08:56:45AM +0100, Rebecca N. Palmer wrote:
> > > > > tensorflow 1.10 was packaged in experimental, but with reduced
> performance,
> > > > > and was removed because this was considered not worth it:
> > > > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=935769
> > > >
> > > > Ah I simply forgot this.  Thanks for refreshing my mind.
> > >
> > > After that I tried to refresh the packaging and uploaded tensorflow 2.0
> > > to the NEW queue. The ftp-master complained about the embedded snapshot
> > > version of Eigen3. Ftp-masters are not convinced even if I said
> > > tensorflow FTBFS against the version shipped in our archive, and it
> > > could waste lots of my time and energy to patch the related code.
> >
> > I can understand your feeling.  I've re-read the discussion[1] about
> > including eigen3 into the tensorflow source.  The most promising
> > statement was given in
> >
> >
> https://alioth-lists.debian.net/pipermail/debian-science-maintainers/2020-March/079169.html
> >
> >Sean Whitton spwhitton at spwhitton.name
> >Tue Mar 3 04:04:52 GMT 2020
> >
> >Rejected per your request, but from my point of view this discussion
> is
> >not over -- Policy 4.13 says that packages *should* not use
> convenience
> >copies of code, not that they *must* not.
> >
> >Thank you for all your work on uploading useful ML packages.
> >
> > At first: Thanks also from me!
> >
> > >From my point of view that is not a lost case so we should really try
> > again.
> >
> > > I was so angry at that time so I deleted my name from the maintainers
> > > and wrote the git message "I'm dropping this burden.".
> >
> > Well, sometimes personal feelings are dominating our actions.  I hope we
> > could form some real team around this to spread the technical as well as
> > the organisational burden.
> >
> > > So another kind notice for whoever is willing to take over tensorflow:
> > > also be prepared to fuss with ftp-master.
> >
> > I need to admit that I'm absolutely happy about ftpmaster.  They are
> > currently *extremely* supportive to our COVID-19 hackathon and > 30
> > packages made it from upload to unstable in less than 24 hours!
> >
> > Since I consider it a "promising time" to reach a lot for deep learning
> > tools in Debian which are frequently used in those tools to hunt down
> > COVID-19 I'd like to call for help here for trying again to get
> > tensorflow in - this time even with the Python3 module.
> >
> > Lumin has written an own build system for the C++ library since he has
> > found out that the upstream build system is not usuable for Debian.
> > Regarding the Python3 module he said that its not simply a matter of
> > adapting his build system but "significantly extend" it since the python
> > building process is much more complicated than the process for C++.
> >
> > Any takers for this task?
> >
> > Kind regards
> >
> >Andreas.
> >
> >
> > [1]
> https://alioth-lists.debian.net/pipermail/debian-science-maintainers/2020-March/079054.html
> >
> > --
> > http://fam-tille.de
> >
>

Re: Missing dependancies for streamlit

[Michael Crusoe]

https://salsa.debian.org/ci-team/autopkgtest/raw/master/doc/README.package-tests.rst

In general, tests are also allowed to access the internet. As this
> usually makes tests less reliable, this should be kept to a minimum; but
> for many packages their main purpose is to interact with remote web
> services and thus their testing should actually cover those too, to
> ensure that the distribution package keeps working with their
> corresponding web service.
>
> Debian's production CI infrastructure allows unrestricted network
> access, in Ubuntu's infrastructure access to sites other than
> `*.ubuntu.com` and `*.launchpad.net` happens via a proxy (limited to
> DNS and http/https).
>

On Tue, Apr 7, 2020 at 12:46 PM Andreas Tille  wrote:

> On Tue, Apr 07, 2020 at 11:15:17AM +0100, Rebecca N. Palmer wrote:
> > A potential workaround for testing: while package builds (including
> > build-time tests) are not allowed to use the network, autopkgtests *are*
> > allowed to => skip the tensorflow-needing tests in build, but run them
> (with
> > tensorflow from PyPI) in autopkgtest?
>
> Are you sure that autopkgtests *are* allowed to?  That's new to me.
>
> Kind regards
>
>  Andreas.
>
> --
> http://fam-tille.de
>
>

-- 
Michael R. Crusoe

Second Conference of Research Software Engineers: Sep 07-08, 2017

[Michael Crusoe]

Hello all,

As you all probably know, there is a huge need for sustainable community
maintenance of research software; and I think Debian is the logical
foundation for such efforts due to its distributed nature and extensive
technical & social infrastructure**

To help make this a reality I urge any Debian contributor who cares about
science to participate in this year's Conference of Research Software
Engineers:

http://www.rse.ac.uk/conf2017.html

Last year's program: http://www.rse.ac.uk/conf2016_programme

** and shadow infrastructure!

Cheers,

-- 
Michael R. Crusoe
Community Engineer & Co-founder
Common Workflow Language project 
https://impactstory.org/u/-0002-2961-9670
michael.cru...@gmail.com
+1 480 627 9108
+32 2 808 25 58

Re: Re: Location for CWL tool descriptions

[Michael Crusoe]

Hello Steffen,

Thank you for your support. I would recommend that local users can use
/usr/local/share/cwl for site-wide non-Debian installed software and
 ${XDG_DATA_HOME}/cwl/${binary-name}.cwl for per-user installed apps.

On Mon, Jul 6, 2015 at 11:17 AM Steffen Möller steffen_moel...@gmx.de
wrote:

 Hello,

 I am happy to see this. I tend to agree that /usr/share/cwl sounds like
 the right place for everything coming through upstream.

 To accomodate for local user (or Debian package maintainer) additions,  I
 suggest to also allow for /etc/cwl.d/ .  Some
 /etc/cwl.conf may have an option to support any additional set of folders.

 Best,

 Steffen


  Gesendet: Sonntag, 05. Juli 2015 um 19:03 Uhr
  Von: Andreas Tille andr...@an3as.eu
  An: debian-...@lists.debian.org, Debian Science List 
 debian-science@lists.debian.org
  Betreff: Re: Location for CWL tool descriptions
 
  Hi,
 
  I think Michael's proposal is sensible but I think Debian Science
  should be informed as well - so forwarding to this list.
 
  Kind regards
 
 Andreas.
 
  On Sun, Jul 05, 2015 at 03:34:34PM +, Michael Crusoe wrote:
   Hello Debian Med team,
  
   The Common Workflow Language [CWL] working group has created a portable
   method to describe the command line interface of non-interactive
   (scientific) computing tools.
  
   Ideally tool authors would write and ship such descriptions with their
   tools. In suggesting that they do so we need to provide advice as to
 where
   to install said files.
  
   I propose that such descriptors be installed into
   /usr/share/cwl/${binary-name}.cwl
  
   For applications not installed site-wide I propose that all CWL tool
   descriptions should go to $XDG_DATA_HOME/cwl/${binary-name}.cwl
   $XDF_DATA_HOME is from the XDG Base Directory Specification [XDGBDS];
 as
   per that standard it should be interpreted as $HOME/.local/share if not
   defined.
  
   What do people think? By my reading this is compliant with the Debian
   Policy Manual but I am happy to hear other suggestions or corrections.
  
   [CWL] http://common-workflow-language.github.io
   [XDGBDS]
 http://standards.freedesktop.org/basedir-spec/basedir-spec-0.6.html
  
   Cheers,
  
   --
   Michael R. Crusoe
   --
   Michael R. Crusoe: Programmer  Bioinformatician cru...@ucdavis.edu
   The lab for Data Intensive Biology; University of California, Davis
   https://impactstory.org/MichaelRCrusoe http://twitter.com/biocrusoe
 
  --
  http://fam-tille.de
 
 
  --
  To UNSUBSCRIBE, email to debian-med-requ...@lists.debian.org
  with a subject of unsubscribe. Trouble? Contact
 listmas...@lists.debian.org
  Archive: https://lists.debian.org/20150705170308.gb5...@an3as.eu
 
 


 --
 To UNSUBSCRIBE, email to debian-med-requ...@lists.debian.org
 with a subject of unsubscribe. Trouble? Contact
 listmas...@lists.debian.org
 Archive:
 https://lists.debian.org/trinity-54451669-8713-4c71-9729-307484147e46-1436177814539@3capp-gmx-bs37

 --
Michael R. Crusoe: Programmer  Bioinformatician cru...@ucdavis.edu
The lab for Data Intensive Biology; University of California, Davis
https://impactstory.org/MichaelRCrusoe http://twitter.com/biocrusoe