Re: Bug#1066871: RFS: libhyprlang/0.5.0-1 [ITP] -- Configuration language for Hyprland
Hi Alan, I granted you with the maintainer access to this repo: https://salsa.debian.org/debian/hyprlang This package has cleared the NEW queue a while ago: https://tracker.debian.org/pkg/hyprlang Could you please push your changes from personal repo to the above repo? I can also do it for you if you don't mind not being the git committer. I agree with splitting these packages for the long run. Will create repos for other packages and invite you as well. Does it sound good to you? Repos under the public debian/ namespace allows other people to help without much permission issues. On 3/14/24 16:36, Alan M Varghese wrote: Hello Mo, May I address you Mo? I am happy to co-maintain hyprland with you. :) The ITP for hyprland[0] was created by werdahias@ who had created an initial skeleton for the packaging a while ago. Under his advise, I decided to de-vendor all of udis86, tracy and hyprland-protocols. As far as I understand, the Debian policy recommends de-vendoring over including files from other sources. I have been working on this for a while and just uploaded them all to mentors and created RFS for them. Currently I have completed packaging hyprland and all its dependencies to the best of my ability. Regarding the points you shared: 1. I wasn't sure what to do with tracy. I have however de-vendored it and created an RFS for it[1]. But, I am unable to get the GPU traces working on my AMD RX 6600 (for a debug build of Hyprland with tracy enabled). I am not sure if this is because of my device or something else. I have seen some discussion upstream that tracy's GPU traces are not always reliable. Tracy seems to work fine, otherwise. 2. I have de-vendored udis86 too. The library and the included CLI seems to run fine. Here is the RFS[2]. 3. Again, I have separated hyprland-protocols and the RFS is here[3]. You can find the VCS for all hyprland related stuff I did, under the NyxTrail namespace in salsa[4]. The packages all seem to run fine so far. This is my first time packaging for Debian and any feedback is welcome. Let me know how you wish to proceed. Regards, Alan [0] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1040971 [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1066873 [2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1066870 [3] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1066868 [4] https://salsa.debian.org/NyxTrail On 3/15/24 01:10, Mo Zhou wrote: Hi Alan, Thank you for your work! I did not check the ITP bugs before we make overlapping efforts: https://salsa.debian.org/debian/hyprlang https://salsa.debian.org/debian/hyprland I just rushed the two packages within a short time the last night. They work properly on Sid with my laptop. I have uploaded hyprlang to NEW without checking ITP https://ftp-master.debian.org/new/hyprlang_0.5.0-1~exp1.html The hyprland is still pending as I've not yet finished the debian/copyright part. In terms of build depends of hyprland: 1. tracy is optional. USE_TRACY is by default off. We can build the package without tracy. 2. the udis86 is embedded in the upstream tarball as well. Maybe we can keep it embedded as udis86 is only needed by hyprland 3. hyprland-protocols is also embedded. I suppose it is ok to keep this specific project, instead of splitting the package to increase the required amount of work. Shall we merge our work and co-maintain this? On 3/14/24 14:46, Alan M Varghese wrote: Package: sponsorship-requests Severity: wishlist X-Debbugs-Cc: a...@digistorm.in Dear mentors, I am looking for a sponsor for my package "libhyprlang": * Package name : libhyprlang Version : 0.5.0-1 Upstream contact : vaxerski * URL : https://github.com/hyprwm/hyprlang * License : LGPL-3+ * Vcs : https://salsa.debian.org/NyxTrail/hyprlang Section : x11 The source builds the following binary packages: libhyprlang2 - Configuration language for Hyprland (library) libhyprlang-dev - Configuration language for Hyprland (development files) To access further information about this package, please visit the following URL: https://mentors.debian.net/package/libhyprlang/ Alternatively, you can download the package with 'dget' using this command: dget -x https://mentors.debian.net/debian/pool/main/libh/libhyprlang/libhyprlang_0.5.0-1.dsc Changes for the initial release: libhyprlang (0.5.0-1) UNRELEASED; urgency=low . * Initial release. Closes: #1065352 Regards,
Re: Bug#1066871: RFS: libhyprlang/0.5.0-1 [ITP] -- Configuration language for Hyprland
Hi Alan, Thank you for your work! I did not check the ITP bugs before we make overlapping efforts: https://salsa.debian.org/debian/hyprlang https://salsa.debian.org/debian/hyprland I just rushed the two packages within a short time the last night. They work properly on Sid with my laptop. I have uploaded hyprlang to NEW without checking ITP https://ftp-master.debian.org/new/hyprlang_0.5.0-1~exp1.html The hyprland is still pending as I've not yet finished the debian/copyright part. In terms of build depends of hyprland: 1. tracy is optional. USE_TRACY is by default off. We can build the package without tracy. 2. the udis86 is embedded in the upstream tarball as well. Maybe we can keep it embedded as udis86 is only needed by hyprland 3. hyprland-protocols is also embedded. I suppose it is ok to keep this specific project, instead of splitting the package to increase the required amount of work. Shall we merge our work and co-maintain this? On 3/14/24 14:46, Alan M Varghese wrote: Package: sponsorship-requests Severity: wishlist X-Debbugs-Cc: a...@digistorm.in Dear mentors, I am looking for a sponsor for my package "libhyprlang": * Package name : libhyprlang Version : 0.5.0-1 Upstream contact : vaxerski * URL : https://github.com/hyprwm/hyprlang * License : LGPL-3+ * Vcs : https://salsa.debian.org/NyxTrail/hyprlang Section : x11 The source builds the following binary packages: libhyprlang2 - Configuration language for Hyprland (library) libhyprlang-dev - Configuration language for Hyprland (development files) To access further information about this package, please visit the following URL: https://mentors.debian.net/package/libhyprlang/ Alternatively, you can download the package with 'dget' using this command: dget -x https://mentors.debian.net/debian/pool/main/libh/libhyprlang/libhyprlang_0.5.0-1.dsc Changes for the initial release: libhyprlang (0.5.0-1) UNRELEASED; urgency=low . * Initial release. Closes: #1065352 Regards,
Re: Policy: should libraries depend on services (daemons) that they can speak to?
On 1/7/24 15:36, Jonas Smedegaard wrote: Quoting Ansgar (2024-01-07 20:39:57) I therefore think that libraries (be it classic C shared object libraries or Python modules or others) should in general *not* have Depends: or Recommends: relations on services (DBus services, DBus itself, daemons, ...). I thought this was already in policy. If not then yes, certainly makes sense to add that! Regarding "I thought ... if not ...", please allow me to jump in and advertise a little bit. $ debgpt -H -f policy_index.txt -A 'According to this table of contents, which part may have discussed dependency of shared objects on services which they can speak to?' --openai_model gpt-3.5-turbo-16k LLM [2]> Based on the table of contents, the part that may have discussed the dependency of shared objects on services they can speak to is section 8.6, titled "Dependencies between the library and other packages." This section covers the dependencies between the packages of the same library and the dependencies between the library and other packages. It may include information on how shared objects can communicate with services they depend on. $ debgpt -H --policy 8.6 -A 'does this section discuss that shared object packages should generally *NOT* Depends: or Recommends: services which they speak to?' --openai_model gpt-3.5-turbo-16k LLM [2]> No, this section does not discuss whether shared object packages should or should not have dependencies on services they speak to. Section 8.6 of the Debian Policy specifically focuses on dependencies between the library and other packages. It explains the mechanisms of using "symbols" files or "shlibs" files to determine the required dependencies for shared libraries. It also provides guidelines for maintaining shared library packages and handling backward-compatible ABI changes. lumin[3]> OK. Where should I add it if I would like to introduce this new policy? LLM[4]> If you would like to introduce a new policy regarding shared object packages and their dependencies on services, you would need to propose the change to the Debian Policy mailing list. The mailing list is the appropriate forum for discussing and proposing changes to the Debian Policy. Once your proposal is accepted and approved by the Debian Policy team, it will be incorporated into the official Debian Policy document. ...
Re: DebGPT: how LLM can help debian development? demo available.
debgpt v0.4.90 has been uploaded to NEW, targeting at unstable. This tool is still under development, new features will be added later. Usage examples can be found in debgpt(1) or README.md . They are the same file. During my (limited number) of experiments when developing this tool, LLMs are good at: 1. $ debgpt git commit (many recent git commits in debgpt repo are generated using LLM) This is the one I like most so far. This will likely become a part of my git workflow. 2. summarizing arbitrary texts in pretty format 3. explaining arbitrary code files 4. fortune, e.g. $ debgpt fortune :joke 5. writing boring boiler plate code (such as matplotlib) LLMs are bad at: 1. logic and deduction for example, it does not always tell the relation between "SyntaxError: print 'hello world'" and an upstream PR named "initial python3 support. (Maybe chain of thoguht can improve this but I postponed it) 2. generating unix-format patches Let it give you the edited files directly, instead of the diff. This tool is not specific to Debian distribution. Adding additional text sources like Arch Wiki, Gentoo Policy/Wiki is very easy and planned as future feature. Or you can load these web pages through `debgpt --cmd 'curl '`. Support for dealing very long documents will be further investigated in future releases. On 1/3/24 20:07, Mo Zhou wrote: I have implemented the OpenAI API frontend, with streaming to terminal enabled. On 1/2/24 17:07, M. Zhou wrote: Following what has been discussed in d-project in an offtopic subthread, I prepared some demo on imagined use cases to leverage LLMs to help debian development. https://salsa.debian.org/deeplearning-team/debgpt
Re: Bug#1060034: ITP: python-openai -- OpenAI Python API library
On 1/5/24 12:32, Sam Hartman wrote: Also, I thought that there were several open-source implementations of this API, including from llama.cpp and ctransformers among others. My impression was the OpenAI API had become kind of a standard for gluing something like text-generation-webui to a hosted model not running locally. Or is that some *different* OpenAI API? You are correct. The OpenAI API spec is already a de-facto standard in this area. Every alternative will try to provide the same set of API.
Re: Bug#1060034: ITP: python-openai -- OpenAI Python API library
On 1/5/24 11:45, Ansgar wrote: Then the package should be in main. We do not require external software to be free as well, be that Web APIs provided by Github, Twitter, or the NVidia firmware required for Nouveau, microcode or storage/keyboard/sound/printer firmware required for Linux, ... We would have to move pretty much everything to contrib if that was the case. OK. It seems that I misunderstood it all the time. Mailed ftp-master to reject so I can change the section and reupload. That means src:debgpt can go to the main section as well.
Re: Bug#1060034: ITP: python-openai -- OpenAI Python API library
On 1/5/24 03:48, Ansgar wrote: On Thu, 2024-01-04 at 21:30 -0500, Mo Zhou wrote: Dependency of DebGPT. Will be maintained by deep learning team. It will go to the contrib section based on policy section 2.2.2, because this API client requires either (1) paied access to the proprietary backend (2) compatible open-source backend implementations are not yet available in the archive. I'm confused. Will this package have a "Depends: some-proprietary- openai-thing"? Can't it talk to a web service providing the API? No. All the dependencies are: python3-anyio,python3-distro, python3-httpx,python3-pydantic,python3-sniffio,python3-tqdm, python3-typing-extensions,python3:any. Can be satisfied using packages from main section. That said, this package does not work at all without (1) paid access token from OpenAI (2) open-source alternative https://github.com/lm-sys/FastChat/blob/main/docs/openai_api.md One just need to export OPENAI_BASE_URL or set it in program as openai.base_url = '...' to switch the server to connect. It is a client API package, and its compatible Does OpenAI even offer this "some-proprietary-openai-thing" package to the general public? No. They provide nothing else than this API client.
Bug#1060034: ITP: python-openai -- OpenAI Python API library
Package: wnpp Severity: wishlist Owner: debian X-Debbugs-Cc: debian-devel@lists.debian.org * Package name : python-openai Version : 1.6.1 * URL : https://github.com/openai/openai-python * License : Apache-2.0 Programming Lang: Python Description : OpenAI Python API library Dependency of DebGPT. Will be maintained by deep learning team. It will go to the contrib section based on policy section 2.2.2, because this API client requires either (1) paied access to the proprietary backend (2) compatible open-source backend implementations are not yet available in the archive. Thank you for using reportbug
Re: DebGPT: how LLM can help debian development? demo available.
I have implemented the OpenAI API frontend, with streaming to terminal enabled. Just export your OPENAI_API_KEY to environment if you have one, and specify `-F openai` in the debgpt command line. It work work without the self-hosted LLM inference backend. That means the command `debgpt none -i -F openai` falls back to a general terminal client to ChatGPT. I believe adding this frontend allows more people to try LLMs with debian-specific tasks conveniently. Since the openai frontend allows people to use this tool without pytorch and a bunch of deep learning frameworks, I plan to upload debgpt to experimental/non-free, following python-openai (unstable/non-free). Packages in debian archive should be enough to run the openai frontend, but not for the self-hosted LLM backend. The following are my replies to previous comments: On 1/2/24 17:49, Jeremy Stanley wrote: but one of the most useful ways I've seen LLM leveraged is large open source projects feeding in their corpus of complex documentation, and then providing users with a human language interaction prompt where they can ask questions and get conversational responses based on that documentation. A continuously trained LLM backing a sort of "search" function for all of www.d.o/doc (and maybe also wiki.d.o) could be cool. Yes. So the `debgpt policy ...` and `debgpt devref` are two examples on letting LLM read long documents and answer questions. The problem is that the full document is too long, while the supported context length is just 4k ~ 16k tokens for openai api, or 8k tokens for self-hosted mistral7b. Through merely prompt engineering, of course we cannot feed the whole policy document to the context. Because solely section 4.9.1 is already overlength against the typical chatgpt model gpt-3.5-turbo. That's why the interface is designed to feed a specific section of long document. That said, with more works, I think it should be possible to feed the LLM the table of contents first, and let it choose a section it wants to refer based on your question. On 1/3/24 02:58, Andrius Merkys wrote: I find this pretty impressive. Thanks a lot for working on it. Thanks. I had fun experimenting with this. To me the most time consuming task in Debian recently is the Python transitions. I wonder whether DebGPT could help with them. Maybe there are other, non-Debian-specific GPTs for this task, but I would prefer a Debian one. This is in fact not Debian specific. Essentially the current status of this project is almost a prompt generator, automatically gathering information about the task you specified in the command line, and send all the information to LLM. It is not different from using the web-based ChatGPT, by copy and pasting the same information before asking the question to LLM. But if the requirement is relatively loose -- it can be seen as Debian specific. Particularly for ChatGPT, there is a debian-specific system prompt in debgpt/frontend.py, which asks ChatGPT to play the role of a Debian developer when responding. On 1/3/24 04:33, Andrius Merkys wrote: Mostly failing tests, and mostly due to API changes between subsequent Python 3.x versions. My idea is to extract the failure from the BTS, and append the "breaking changes" section of the cpython changelog, and see what suggestion the LLM can provide. I do not expect perfect bug fixing suggestions but it should be able to conclude something. The corresponding cli may look like this: $ debgpt bts --id BUG_NUMBER --pychange 3.11-3.12 free -i # to be implemented Or possily pull the list of recent github upstream issues list, and let LLM figure out which upstream bug or pull request is most relevant. $ debgpt bts --id BUG_NUMBER --upstream_issues free -i # to be implemented The upstream issues webpage can be found in the Control/metadata file, and the source package name can be typically found in the bugs page. This automatic process should be able to save people some time. On 1/3/24 07:06, Andrey Rakhmatullin wrote: So the solution is either find a patch in the upstream repo (committed or proposed in issues/PRs) or write one yourself. Not sure what can AI help here with. We can ask LLMs to suggest a fix to the bug. Or let LLM to check whether the recent upstream issues list contains a title which might be relevant to the content of the debian bug report. For instance, the LLM can tell `Syntax error: , print "hello world"` is most relevant to an upstream pull request named, e.g. `[pull-request] initial python3 support`, and directly give you the pull request link. (you may have to retry a couple of times to make it think correctly though). I made up a test sample about this at: examples/e4d7fc9f-7469-4cad-959d-373b89498663-initmsg.txt Complicated inference tasks are what we hope LLMs can do very well eventually. Grasping the semantics of natural language is something traditional software can never do. On 1/3/24 03:09, PICCA Frederi
Recalling Key Points of the Previous Attempt (was: Re: Regarding the new "Debian User Repository"
Hi fellow devs, As the previous attempt on debian user repository has been mentioned, I'm interested in recalling some key points of it and bits I've learned, along with some updates. # how the initial attempt started In terms of debian-related works, I mainly focus on scientific computing and deep learning packages. These packages are very performance-sensitive. For example, the performance of BLAS / LAPACK significantly impacts the performance of the whole inverse dependency tree including important nodes such as Numpy, Octave, etc. [8] Namely, a fast BLAS / LAPACK implementation benefits the whole tree wherever there is mathematical operations on contiguous numerical arrays. However, tensorflow is an exception, as it is built upon eigen3 (a header only linear algebra library in C++) instead of BLAS/LAPACK. Eigen3 does not support dynamic code branch selection based on CPU capability (SSE/AVX/etc.), and hence performs very poor if compiled against Debian's default AMD64 ISA baseline. There are a few methods to bump the ISA baseline for a debian package for the official archive: (1) patch the code with gcc's fmv feature; (2) use the "hardware capabilities" feature of ld.so(8); (3) let the user modify debian/rules and rebuild package locally; (4) directly bump the ISA baseline for the whole archive; (5) Gentoo-style source-based partial Debian distribution. For (1), it's impossible to patch tensorflow (millions of lines of code); Solution (2) will result in very bulky binary packages; Solution (3) is somewhat convincing to me since I think a serious user who need performance should learn to build optimized software (e.g. -march=native); Solution (4) was implemented as SIMDebian (deprecated). Solution (5) is implemented as the previous attempt of Debian User Repository (deprecated). I shall briefly review solution (4) and (5) in following sections [8] If you are concerned about the BLAS/LAPACK performance on Debian: https://wiki.debian.org/DebianScience/LinearAlgebraLibraries # regarding SIMDebian The core idea is a partial Debian archive (binary packages with bumped ISA baseline) containing some selected packages which manifests pronounced performance gain [9]. My implementation modifies the default buildflags of dpkg [10], and hence is able to inject "-march=xxx" into the package building process without modifying the source of a debian package. Thus we can automatically rebuild an optimized partial debian archive. My knowledge on which package could benefit from this is rather limited -- as far as I know only Eigen3 reverse dependencies do. We may borrow some experience from the Gentoo community but I eventually stopped experimenting this idea, because there are still some other issues that this idea cannot deal with. [9] https://github.com/SIMDebian/SIMDebian (discontinued) [10] https://github.com/SIMDebian/dpkg/commit/13b062567ac58dd1fe5395fb003d6230fd99e7c1 # regarding previous attempt on DUR What if we just distribute packaging scripts and let users build the packages on their localhost like Gentoo? In this way inserting "-march=native" in d/rules can be sensible in many cases. The problematic license of Nvidia's non-free blob (e.g., cuDNN which is inevitable for most deep learning users) can be bypassed if it is downloaded and locally-packed on the user's host. Apart from deep learning, the CUDA version of ffmpeg is useful for multimedia users, but can only be built locally from source due to license. The license issue of ToxicCandy [12] deep learning models can be bypassed if the user wants to take their own risks and install some fancy AI toys. Things can be more complicated if we combine an automatic code generator [13] and the GPL license. Anyway, Gentoo-style source-based user repository, allows the legal issues to be offloaded to the endusers who are willing to accept them [14]. [11] https://github.com/dupr/duprkit/blob/master/doc/motivation.md [12] https://salsa.debian.org/deeplearning-team/ml-policy/-/blob/master/ML-Policy.rst [13] https://copilot.github.com/ [14] I accept Nvidia's license as a personal user for deep learning purpose. But I tend to refuse Nvidia's license when working for Debian ... Bypassing the license issue is the only viable way I can think of If we want to integrate more fancy AI stuff into the system in the future. # summary and further discussion SIMDebian tries to bump the ISA baseline and create a binary partial Debian archive. The previous attempt of DUR tries to only distribute packaging scripts to the enduser and let the users build packages locally, and hence simultaneously achieving package optimization, and legal issue circumvent. I find my proposed concept "ToxicCandy Model" rather interesting. Identifying such models can really prevent something tricky from sneaking into our archive. We shall see how it works in the future. Dealing with deep learning for Debian is basically a headache -- we need performance, but we are
Bug#977344: ITP: orchis-gtk-theme -- a Material Design theme for GNOME/GTK based desktop environments
Package: wnpp Severity: wishlist Owner: Mo Zhou X-Debbugs-Cc: debian-devel@lists.debian.org * Package name: orchis-gtk-theme Version : git master Upstream Author : https://github.com/vinceliuice/ * URL : https://github.com/vinceliuice/Orchis-theme * License : GPL-3.0 Programming Lang: CSS Description : a Material Design theme for GNOME/GTK based desktop environments Generally I don't tweak my desktop environment given a decent default, but this GTK theme is so beautiful that it temporarily changed my mind. It looks pretty under Gnome3, while it's a pity that it does not work under MATE. This package will be maintained under a salsa team where gtk themes are relevant.
Bug#976805: ITP: imhex -- Hex Editor for Reverse Engineers
Package: wnpp Severity: wishlist Owner: Mo Zhou X-Debbugs-Cc: debian-devel@lists.debian.org * Package name: imhex Version : 1.5.0 * URL : https://github.com/WerWolv/ImHex * License : GPL-2.0 Programming Lang: C++ Description : Hex Editor for Reverse Engineers This will be maintained under security-tools team. (applying salsa access shortly)
Bug#975369: ITP: faiss -- library for efficient similarity search and clustering of dense vectors
Package: wnpp Severity: wishlist Owner: Mo Zhou X-Debbugs-Cc: debian-devel@lists.debian.org, debian...@lists.debian.org * Package name: faiss Version : 1.6.4 Upstream Author : facebook research * URL : http://www.example.org/ * License : MIT Programming Lang: C++, Python Description : library for efficient similarity search and clustering of dense vectors Will be maintained by deep learning team.
Bug#973341: ITP: libnop -- libnop: C++ Native Object Protocols
Package: wnpp Severity: wishlist Owner: Mo Zhou X-Debbugs-Cc: debian-devel@lists.debian.org, debian...@lists.debian.org * Package name: libnop Upstream Author : google * URL : https://github.com/google/libnop * License : Apache-2.0 Programming Lang: C++ Description : libnop: C++ Native Object Protocols PyTorch 1.7.0 build-depends on tensorpipe snapshot XXX; tensorpipe snapshot XXX build-depends on libnop snapshot YYY. I hold an neutral opinion towards the invention of `git submodule`, as long as there is no recursion and force-pushes.
Bug#973002: ITP: tensorboard -- TensorFlow's Visualization Toolkit
Package: wnpp Severity: wishlist Owner: Mo Zhou X-Debbugs-Cc: debian-devel@lists.debian.org, debian...@lists.debian.org * Package name: tensorboard Version : 2.3.0 Upstream Author : TensorFlow authors * URL : https://github.com/tensorflow/tensorboard * License : Apache-2.0 Programming Lang: TypeScript, Python Description : TensorFlow's Visualization Toolkit TensorFlow's visualization tool can be used standalone (i.e. without tensorflow). It's very useful for training networks using PyTorch as well. This package will be maintained by Debian Deep Learning Team for sake of unblocking the pytorch-lightning package.
Bug#972804: ITP: yyjson -- A high performance JSON library written in ANSI C.
Package: wnpp Severity: wishlist Owner: Mo Zhou X-Debbugs-Cc: debian-devel@lists.debian.org * Package name: yyjson Upstream Author : https://github.com/ibireme * URL : https://github.com/ibireme/yyjson * License : MIT/X Programming Lang: C Description : A high performance JSON library written in ANSI C. simdjson upstream plans to add yyjson as a fallback in 0.7 release. https://github.com/simdjson/simdjson/issues/1245 will be maintained under salsa.d.o/debian/yyjson
Bug#972687: ITP: pytorch-lightning -- lightweight PyTorch wrapper for high-performance AI research
Package: wnpp Severity: wishlist Owner: Mo Zhou X-Debbugs-Cc: debian-devel@lists.debian.org, debian...@lists.debian.org * Package name: pytorch-lightning Version : 1.0.0 * URL : https://github.com/PyTorchLightning/pytorch-lightning * License : Apache-2.0 Programming Lang: Python Description : lightweight PyTorch wrapper for high-performance AI research I like this abstraction layer of pytorch. This package will be maintained by Debian Deep Learning Team.
Bug#971692: ITP: openzfs-docs -- OpenZFS Documentation
Package: wnpp Severity: wishlist Owner: Mo Zhou X-Debbugs-Cc: debian-devel@lists.debian.org * Package name: openzfs-docs Version : waiting for the first release Upstream Author : OpenZFS authors * URL : https://github.com/openzfs/openzfs-docs * License : CC-BY-SA Programming Lang: python Description : OpenZFS Documentation Will be maintained by the ZFS on Linux team.
Bug#971650: ITP: ncnn -- high-performance neural network inference framework optimized for the mobile platform
Package: wnpp Severity: wishlist Owner: Mo Zhou X-Debbugs-Cc: debian-devel@lists.debian.org, debian...@lists.debian.org * Package name: ncnn * URL : https://github.com/Tencent/ncnn * License : BSD-3-Clause Programming Lang: C++ Description : high-performance neural network inference framework optimized for the mobile platform I'm intrigued by this framework due to its vulkan support -- another route of hardware acceleration -- and the vulkan dependencies are already present in our archive -- this is a short path. Will be maintained by Debian Deep Learning Team.
Bug#970575: ITP: pytorch-geometric -- Geometric Deep Learning Extension Library for PyTorch
Package: wnpp Severity: wishlist Owner: Mo Zhou X-Debbugs-Cc: debian-devel@lists.debian.org * Package name: pytorch-geometric * URL : https://github.com/rusty1s/pytorch_geometric * License : MIT/X Programming Lang: Python Description : Geometric Deep Learning Extension Library for PyTorch Debian Deep Learning Team
Bug#968426: ITP: tensorpipe -- A tensor-aware point-to-point communication primitive for machine learning
Package: wnpp Severity: wishlist Owner: Mo Zhou X-Debbugs-Cc: debian-devel@lists.debian.org * Package name: tensorpipe Upstream Author : pytorch developers * URL : https://github.com/pytorch/tensorpipe * License : BSD Programming Lang: C++ Description : A tensor-aware point-to-point communication primitive for machine learning New PyTorch dependency. required by pytorch 1.6.0
Bug#968219: ITP: openpbs -- An HPC workload manager and job scheduler for desktops, clusters, and clouds.
Package: wnpp Severity: wishlist Owner: Mo Zhou X-Debbugs-Cc: debian-devel@lists.debian.org * Package name: openpbs Version : 20.0.1 Upstream Author : Name * URL : http://www.example.org/ * License : AGPL-3 Programming Lang: C, python, shell Description : An HPC workload manager and job scheduler for desktops, clusters, and clouds. I'm wondering why it is absent in debian.
Bug#964543: ITP: pymc3 -- Bayesian statistical modeling and Probabilistic Machine Learning
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: pymc3 * URL : https://github.com/pymc-devs/pymc3 * License : Apache-2.0 Programming Lang: Python Description : Bayesian statistical modeling and Probabilistic Machine Learning Debian Deep Learning Team.
Re: debdocker - A Debian docker-based personal builder
Hi Samo, I'm insterested in its differences compared to the following existing docker-based builders: 1. debocker https://people.debian.org/~tomasz/debocker.html 2. whalebuilder https://www.uhoreg.ca/programming/debian/whalebuilder And there is a systemd-nspawn-based builder too: 3. debspawn https://github.com/lkorigin/debspawn All the three builders are avaialable in debian sid. On Sat, Jun 27, 2020 at 12:08:30PM +0200, Samo Pogačnik wrote: > Hi, > > I am preparing a packaging support tool similar to pbuilder, except that it > uses > docker containers instead of chroot environments. The project is available > here: > https://salsa.debian.org/spog/debdocker > > The tool is very immature, but already useful to me. I would like to hear any > thoughts and comments regarding the tool in general and regarding its user > experience. > > Quick reference (running out of cloned project - no warranties!): > 1. Help: > $ ./debdocker --help [command] > $ man ./share/man/man8/debdocker.8 > > 2. Create initial base and devel docker images for i.e. debian:sid, which uses > 'debootstrap' for base image like in 'pbuilder'. It creates base directory and > archive in the supplied path (i.e. ../ in the example): > $ sudo ./debdocker create debian:sid all ../ http://deb.debian.org/debian > > 3. Get a source package (i.e. in ../ use 'dget -d...). > > 4. Build a package, which creates additional (i.e. debian:sid-package_devel) > docker image with installed package's build deps (it is reused while working > on > the same package): > $ ./debdocker build debian:sid ../package_*.dsc > > 5. You can also build extracted (or cloned) sources providing their path > instead > of the 'dsc_file'. There is also a possibility to enter containers (see the > 'enter' command)... > > 6. Building an initial 'debdocker' debian package is supported in the project. > > thanks, Samo >
Re: Pimp your shell - Debian developer tips?
Hi Otto,
On Wed, May 27, 2020 at 10:06:29PM +0300, Otto Kekäläinen wrote:
> Do we have Debian devs here who have pimped their shell heavily with custom
> prompts, colors, command line fonts, shell window title hacks, perhaps using
> zsh etc? Have you written blogs about you experiences, can you share some good
> reads (with screenshots) of what you have done?
I'm possibly the laziest guy among people who replied to this thread.
I used to heavily customize my bash prompt when I was a bash user. But
immediately after my discovery of fish shell (apt install fish), I
was inclined to choose a extremely simple prompt, by just clicking
the favorite prompt theme in fish's GUI config.
~ ❯❯❯
^ This is the "Sorin" prompt theme of fish.
My zsh prompt is basically same, with an additional right side clock:
PROMPT="%F{cyan}%~%f %F{red}❯%F{yellow}❯%F{green}❯%F{cyan}❯%f%b "
RPROMPT="%B%(?..%F{red}[%?]%f )%F{magenta}%*%f"
Personally I dislike tinkering the prompt alot, because IMHO some
other features such as "syntax highlighting" and "automatic suggestion"
are more important and useful for productivity.
TBH I'm a fish user exactly due to the two features.
When I have to use a POSIX shell, I'll choose zsh and use the following
two plugins:
zsh-autosuggestions
zsh-syntax-highlighting
> I've read a bit on zsh and powerline and the like, but I am annoyed that all
> those blog posts are quite superficial and do not mention security,
> interoperability or performance aspects. Frankly any blog post that recommends
> cloning random repos or even worse, running wget | sh something gives me
> chills.
I don't worry about security. All the features I want can be installed
via apt. I'm also the uploader of some of them.
On the other hand, things like oh-my-zsh are too heavy to me, since I
only rely on highlighting and autosuggestion.
Bug#962028: ITP: python-rich -- Rich is a Python library for rich text and beautiful formatting in the terminal.
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: python-rich * URL : https://github.com/willmcgugan/rich * License : MIT Programming Lang: Python Description : Rich is a Python library for rich text and beautiful formatting in the terminal. will be maintained under DPMT
Bug#961565: ITP: skorch -- A scikit-learn compatible neural network library that wraps PyTorch
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: skorch * URL : https://github.com/skorch-dev/skorch * License : BSD-3 Description : A scikit-learn compatible neural network library that wraps PyTorch Debian Deep Learning Team
Bug#961556: ITP: stan -- a state-of-the-art platform for statistical modeling and high-performance statistical computation
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: stan * URL : https://github.com/stan-dev/stan * License : bsd-3 Description : a state-of-the-art platform for statistical modeling and high-performance statistical computation
Bug#961557: ITP: stan-math -- a C++ template library for automatic differentiation of any order
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: stan-math * URL : https://github.com/stan-dev/math * License : bsd-3 Description : a C++ template library for automatic differentiation of any order
Re: Bug#961029: Request for new mailing list: debian-ai
Control: retitle -1 Request for new mailing list: debian-ai Thanks for the various feedbacks. As pointed out by lamby, the "debian-mlhwaccl" may be quite hard to search. Meanwhile, the name can be overlength if we expand that abbreviation. So I'm proposing to use a new list name: debian-ai instead of the old: debian-mlhwaccl Name: debian-ai Short-Desc: Artificial Intelligence Software Development in Debian Long-Desc: Development of artificial intelligence software, such as machine learning, deep learning, compilers and hardware acceleration software stacks. The topics for this list may include any one of the following (but not limited): * Mahcine Learning: sklearn, xgboost, AutoML, ... * Deep Learning: tensorflow, pytorch, mxnet, cntk, ... * Computer Vision: opencv, skimage, ... * Computational Linguistics: nltk, spacy, ... * Hardware Acceleration: CUDA, ROCm, OpenCL, SYCL, SIMD * Compiler: LLVM (for MLIR, SPIR-V, SYCL), GCC (gcc-offload-nvptx) And sometimes the topics in this list may be complex intersections of the mentioned keywords. Category: development Subscription-Policy: open Post-Policy: open Web-Archive: yes
Bug#961182: ITP: nni -- An open source AutoML toolkit for automate machine learning lifecycle
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: nni * URL : https://github.com/microsoft/nni * License : MIT/Expat Programming Lang: Python Description : An open source AutoML toolkit for automate machine learning lifecycle AutoML. Debian Deep Learning Team.
Bug#961181: ITP: tpot -- Automated Machine Learning tool that optimizes machine learning pipelines using genetic programming
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: tpot * URL : https://github.com/EpistasisLab/tpot * License : LGPL-3.0 Programming Lang: Python Description : Automated Machine Learning tool that optimizes machine learning pipelines using genetic programming AutoML. Debian Deep Learning Team.
Re: RFC: threading-aware virtual BLAS/LAPACK
Hi fellow devs,
I've suddenly got some inspiration on this problem, which resulted in a
much better solution for the problem the original proposal confronts.
I like this overhauled solution.
No extra shared libs, no extra SONAMEs. No extra burden for the
BLAS/LAPACK maintainers. Minor patching work for maintainers with
special demands.
--- New solution ---
1. BLAS providers create new directory under
/usr/lib//libblas-/
and put another copy of their alternative symlinks into the
directories.
e.g. libopenblas-pthread provides the extra symlink:
/usr/lib//libblas-pthread/libblas.so{,.3}
2. Maintainers of reverse dependencies, when they have specific
requirement on the threading implemetation, add the
/usr/lib//libblas-/
directory to the RPATH property of the resulting ELF binaries.
In that way the programs can use the BLAS implementation without
worring about the uncertainty of the alternatives configuration.
Other packages insensitive to the threading implementations will
use the libblas.so.3 in the global scope:
/usr/lib//libblas.so.3 (a symlink)
--- end solution ---
That's it, much simpler and much more efficient than my first version.
Comments please?
On Wed, May 13, 2020 at 10:54:00PM +0200, Gard Spreemann wrote:
> Mo Zhou writes:
>
> > Please comment:
> > 1. Do we have a better solution where we can retain high performance and
> > avoid threading trouble at the same time?
> > 2. If we don't have a better solution, is my proposal acceptable?
> > 3. In which way can my proposal be improved?
>
> Hi Mo,
>
> Your proposal seems sane to be, and a real service to the users!
:-)
> However, I am a little bit worried about the maintenance burden you are
> setting up for the future; may it be promising too much to the users to
> provide every single combination? Since this is BLAS, one could imagine
> adding yet another couple of versions for each of the options, targeting
> ever fancier vector instructions, etc. – at what point are special needs
> best left to the user to cover?
With the second version of solution, we can elegantly solve all the
problem, and prevent the threading trouble from propagating to our end
users!
> I wish I had more constructive criticism. Thanks for the work!
Let me do the critism: the first version of my solution is rubbish ...
> An aside: do autopkgtest or other CIs currently test packages with
> varied combinations of BLAS/LAPACK providers?
No. But that's easy to implement.
For example, we can add the following test cases in the autopkgtest
control file of openblas:
Test-Command: run-numpy-unit-test.sh
Depends: libopenblas-pthread-dev
Test-Command: run-numpy-unit-test.sh
Depends: libopenblas-openmp-dev
Test-Command: run-numpy-unit-test.sh
Depends: libopenblas-serial-dev
In this way we BLAS/LAPACK maintainers can immediately spot regressions
in important packages. (adding these tests seems like my work)
> Best,
> Gard
Bug#961029: Request for new mailing list: debian-mlhwaccl
Package: lists.debian.org Severity: wishlist X-Debbugs-CC: debian-devel@lists.debian.org (I think I should CC -devel since I'm requesting a NEW dev ML) Hi list master, I'd like to request for a new public, and archived mailing list Name: debian-mlhwaccl Short-Desc: "Debian's Machine Learning & Hardware Acceleration" Long-Desc: Development of Machine/Deep Learning Software and Corresponding Hardware Acceleration Measures incl. ROCm, OpenCL, SYCL and CUDA Category: obviously a dev list Subscription-Policy: open Post-Policy: open Web-Archive: yes (we definitely need an archive) Our demand / Rationale: 1. we need an archived mailing list to better coordinate the works related to deep learning / machine learning and hardware acceleration (including ROCm, CUDA, OpenCL, SYCL) specific to machine learning applications. Debian science team maling address is not quite appropriate for the topics. 2. we need an archived mailing list as the maintainer address for the ROCm team to better coordinate the works. We should not abuse debian science team's alioth mailing list address as the maintainer address neither. 3. we need an archived mailing list for discussing about the development of debian deep learning team. The machine learning and deep learing subareas are already significant enough to form a dedicated interest group inside debian. Teams involved: Debian ROCm Team, Debian Deep Learning Team Expected key words for the mailing list: tensorflow, pytorch, bazel, sklearn, opencl, sycl, rocm, cuda, llvm, xla (tensorflow), openmpi as you see, some times the topics could be off-topic to debian-science, and the "machine learning & hardware acceleration" things always involves complicated intersection of various areas. It could be better if we have a dedicated list for the whole thing! thank you very much I'm following https://www.debian.org/MailingLists/HOWTO_start_list
Re: Bug#960981: ITP: rocr-runtime -- HSA Runtime API and runtime for ROCm
On Tue, May 19, 2020 at 08:41:47PM +0900, Norbert Preining wrote: > > Packaging ROCm requires a forked LLVM. Will you package that fork, and > > Thanks for bringing this up. Up to now all the ITP don't require the > forked llvm. We have one repository (rocm-hip) in preparation for which > the llvm-fork is necessary. And there are other's I am preparing. > > Our plan (at the moment basically Mo Zho and me) was to package it into ^ u > a separate directory /usr/lib/llvm/roc/ (like Gentoo does) or into any > similar place. Yes. Aligning with Gentoo looks good, as long as we really need an forked LLVM. Benda Xu (gentoo dev) told me that his latest investigation suggests that AMD's modifications have been upstreamed. But no AMD people responded the question. https://github.com/RadeonOpenCompute/hcc/issues/1421 We'll have to confirm that. > But I am more than open for help with the llvm-roc build, since this not > even finishes on my computer without blowing it up ATM ... My experience is that LLVM build often ends up being killed by the OOM killer, and I can only use -j4 on a machine with 32Gigs of RAM. The "-gsplit-dwarf" CFLAG may help when doing the parellel linking, but I still have not tried it with LLVM yet. > PREINING Norbert https://www.preining.info > Accelia Inc. + IFMGA ProGuide + TU Wien + JAIST + TeX Live + Debian Dev > GPG: 0x860CDC13 fp: F7D8 A928 26E3 16A1 9FA0 ACF0 6CAC A448 860C DC13
Re: Bug#960981: ITP: rocr-runtime -- HSA Runtime API and runtime for ROCm
Hi doko, I've not dug into this deep enough by myself. AFAIK, however, AMD seemd to have upstreamed their LLVM bits specific to LLVM https://github.com/RadeonOpenCompute/hcc/issues/1421 https://github.com/RadeonOpenCompute/hcc#deprecation-notice I asked the corresponding Gentoo developer and he confirms that. On Tue, May 19, 2020 at 11:10:48AM +0200, Matthias Klose wrote: > > Packaging ROCm requires a forked LLVM. Will you package that fork, and > > - if yes, how do you plan do maintain the set of shared libraries >provided by the two versions. > > - if no, how do you integrate with the LLVM found in Debian > > Matthias
Bug#960816: ITP: snapraid -- A backup program for disk arrays.
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: snapraid * URL : https://github.com/amadvance/snapraid * License : GPL-3 Programming Lang: C Description : A backup program for disk arrays.
RFC: threading-aware virtual BLAS/LAPACK
Hi fellow devs, I'm asking advice on a system design issue specific to Debian, which is whether we should introduce threading-aware BLAS/LAPACK virtual package. This question is important because it affects a reverse dependency tree with > 1 million popcon. (One week ago I asked for comment in -science team, but I've not got enough feedback yet: https://lists.debian.org/debian-science/2020/05/msg00023.html) --- Background --- BLAS/LAPACK do fundamental linear algebra with dense vectors and matrices, and is very performance sensitive for many of its rdeps. Apart from optimizing the cache access and accelerating computations with architecture specific intrinsics (SIMD), parallelization is also an inevitable way to boost BLAS/LAPACK performance. An optimized BLAS/LAPACK library may choose one of these threading libraries for parallelization: GNU OpenMP (gomp), Intel/LLVM OpenMP (iomp), TBB, Pthread. Based on that problem may appear in a relatively deep dependency tree. Assume that we compiled a scientific program with clang+openmp, which has been linked against libblas.so.3 and libiomp.so.5 at the same time. When the libblas.so.3 provider is the openmp version of OpenBLAS, our program loads libgomp and libiomp at the same time. Undefined behaviour may happen because iomp/gomp has the same set of symbols. Similarly, when pthread and openmp are indirectly used at the same time, undefined consequence may be incurred. e.g. https://lists.debian.org/debian-science/2020/05/msg00031.html where R-4.0.0 calculation speed get ridiculously slow due to the mixed usage of pthread and openmp. The fact is that, some of the rdeps of BLAS/LAPACK prefer openmp, some prefer pthread, while some perfer serial. In that sense, our current libblas.so.3 alternative is threading-unaware: ^ (high priority in alternatives system) | OpenBLAS (pthread) <- libopenblas0, libopenblas0-pthread | OpenBLAS (openmp) | OpenBLAS (serial) | BLIS (openmp) <- libblis3, libblis3-openmp | BLIS (pthread) | BLIS (serial) | Atlas (? I forgot it) | Netlib (serial)<- libblas3 | libblas.so.3 | Intel-MKL (gomp/iomp/tbb) <- libmkl-rt (non-free) v (low priority in alternatives system) By default, the standard serial implementation (Netlib) will be installed to satisfy the "libblas3 | libblas.so.3" dependency, as the libblas.so.3 provider. But this implementation can be >40x slower than the fastest impl. --- RFC / My Tentative Proposal --- * Apart from libblas.so.3, we create more virtual BLAS/LAPACK alternative groups. E.g. libblasp.so.3 for a pthread BLAS, libblaso.so.3 for an openmp BLAS, and libblass.so.3 for a serial BLAS. (The way of implementation is not my point here. -- Let's focus on the Debian system design issue) In that way (1) maintainers who have a specific requirement on threading library can directly require a BLAS implementation with specific threading lib. (2) the threading troubles won't be propagated to our end users again and again. Downsides: (1) currently src:openblas builds 12 versions of OpenBLAS shared object. If we decide to add threading-aware virtual packages, it will build 18 versions of openblas. (2) less debian developers will be able to readily handle our BLAS/LAPACK ecosystem except for Sébastien and me Please comment: 1. Do we have a better solution where we can retain high performance and avoid threading trouble at the same time? 2. If we don't have a better solution, is my proposal acceptable? 3. In which way can my proposal be improved? Thank you in advance. :-) Ack --- This work is supported by GSoC2020.
Bug#960471: ITP: prophet -- Tool for producing high quality forecasts for time series data that has multiple seasonality with linear or non-linear growth.
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: prophet Upstream Author : Facebook * URL : https://github.com/facebook/prophet * License : MIT Programming Lang: Py, R Description : Tool for producing high quality forecasts for time series data that has multiple seasonality with linear or non-linear growth. Time series analysis tool. Will be maintained by debian science team.
Bug#960428: ITP: blingfire -- Bing's Finite State machine and REgular expression manipulation library
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: blingfire Upstream Author : Microsoft * URL : https://github.com/microsoft/BlingFire * License : MIT Programming Lang: C++, python Description : Bing's Finite State machine and REgular expression manipulation library useful as a preprocessing tool for computational linguistics. This library will be maintained by Debian Deep Learning Team.
Bug#959184: ITP: python-peachpy -- x86-64 assembler embedded in Python
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: python-peachpy * URL : https://github.com/Maratyszcza/PeachPy * License : bsd2 Description : x86-64 assembler embedded in Python NNPACK deps NNPACK is pytorch deps (performance booster)
Bug#959183: ITP: python-opcodes -- Database of CPU Opcodes
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: python-opcodes * URL : https://github.com/Maratyszcza/Opcodes * License : BSD-2 Description : Database of CPU Opcodes NNPACK deps NNPACK is a pytorch performance booster Debian deep learining team
Bug#959121: ITP: qnnpack -- Quantized Neural Network PACKage (pytorch performance)
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: qnnpack * URL : https://github.com/pytorch/qnnpack * License : BSD-3 Programming Lang: C++ Description : Quantized Neural Network PACKage (pytorch performance) pytorch performance booster as suggested by upstream debian deep learning team
Bug#959122: ITP: nnpack -- Acceleration package for neural networks on multi-core CPUs
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: nnpack * URL : https://github.com/Maratyszcza/NNPACK * License : BSD-2 Programming Lang: C, C++ Description : Acceleration package for neural networks on multi-core CPUs pytorch performance booster as suggested by upstream debina deep learning team
Bug#959123: ITP: fbgemm -- Facebook GEneral Matrix Multiplication
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: fbgemm * URL : https://github.com/pytorch/FBGEMM * License : BSD-3 Programming Lang: C++ Description : Facebook GEneral Matrix Multiplication pytorch performance booster as suggested by upstream debian deep learning team
Quick Bits: ML-Policy Updates; Recent Progress of Deep Learning Packaging
Hi fellow devs, Recently an increasing about of preprint papers studying COVID-19 with machine leanring techniques have been published, which reminded me of the importance of machine learning techniques. (1) Updates to ML-Policy [1] Sidenote: Projects such as https://github.com/IliasPap/COVIDNet directly posed questons on me, that are exactly what ML-Policy is trying to solve. (2) Updates to Deep Learning Related Packages Sidenote: Some fellow devs of debian-med and debian-science believe that deep learning software are substantially meaningful as a part of the COVID-19 hackathon ... So ... Updates to ML-Policy Just tagged the v0.2 release, but it is still an experimental UNOFFICIAL document. The most significant changes of this document after the last time we discuss about it on this list, is simplification. Detailed Changes: 1. Removed the definition of "Sourceless Model" and "critical tasks". They will make things more complicated. 2. The previous version only discuss about the pre-trained models. In this version we also discuss about the data, and the output of the models. 3. Rewrote the definition of model reproducibility. 4. Introduced a "whitelist" policy for ToxicCandy models. So that we won't kill things like the input methods. 5. Introduced a "Combination" policy in case multiple models are used together. 6. Introduced a "Tainting" policy in case free and non-free stuff are mixed in the same pipeline. The document is longer, actually, compared to the previous version. But the newly introduced ideas follow the convention and common sense to some extent, which should not make them harder to understand. This policy is not complete. Advices are welcome. BTW, as you might have noticed, the ml-policy.git repo has been transferred to deeplearning-team namespace. Updates to Packages --- Status of the debianization of top-2 deep learning frameworks, i.e. tensorflow and pytorch: tensorflow: it is hard to circumvent the only officially supported build system bazel. We have one developer currently working on the bazel packaging. pytorch: Having been greatly motivated by the attitude of pytorch upstream[2] and their quick response to pull requests, I finished packaging the necessary pytorch dependencies in lightning speed and uploaded them to the NEW queue. Currently I'm working on making the upstream build system distro-friendly. Related works are all tracked in[2]. My local builds are very successful. --- [1] https://salsa.debian.org/deeplearning-team/ml-policy/-/blob/master/ML-Policy.pdf [2] https://github.com/pytorch/pytorch/issues/14699 Mo, On behalf of Debian Deep Learning Team
Bug#958893: ITP: pytorch-ignite -- High-level library to help with training neural networks in PyTorch
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: pytorch-ignite * URL : https://github.com/pytorch/ignite * License : BSD-3 Programming Lang: py Description : High-level library to help with training neural networks in PyTorch make pytorch even easier to use debian deep learning team
Bug#958892: ITP: pytorch-vision -- Datasets, Transforms and Models specific to Computer Vision
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: pytorch-vision * URL : https://github.com/pytorch/vision * License : BSD-3 Programming Lang: C++,py Description : Datasets, Transforms and Models specific to Computer Vision a very important extra module for pytorch users debian deep learning team
Bug#958891: ITP: pytorch-text -- Data loaders and abstractions for text and NLP
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: pytorch-text * URL : https://github.com/pytorch/text * License : bsd-3 Programming Lang: py Description : Data loaders and abstractions for text and NLP allows users to better manipulate textual data using pytorch debian deep learning team
Bug#958890: ITP: pytorch-audio -- Data manipulation and transformation for audio signal processing, powered by PyTorch
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: pytorch-audio * URL : https://github.com/pytorch/audio * License : bsd-2 Programming Lang: py Description : Data manipulation and transformation for audio signal processing, powered by PyTorch enables user to manipulate audio data using pytorch debian deep learning team
Bug#958888: ITP: pytorch -- Tensors and Dynamic neural networks in Python with strong GPU acceleration
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: pytorch * URL : https://github.com/pytorch/pytorch * License : BSD-3 Programming Lang: C++, python Description : Tensors and Dynamic neural networks in Python with strong GPU acceleration One of the top-2 deep learning frameworks I've uploaded all the necessary build dependencies to the NEW queue. Now I'm waiting for the upstream to merge my pull requests. will be maintained under Debian Deep Learning Team
Bug#958818: ITP: xnnpack -- High-efficiency floating-point neural network inference operators for mobile, server, and Web
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: xnnpack * URL : https://github.com/google/XNNPACK * License : BSD_3 Programming Lang: C++ Description : High-efficiency floating-point neural network inference operators for mobile, server, and Web optional pytorch dependency debian deep learning team
Bug#958655: ITP: pthreadpool -- pthread-based thread pool for C/C++
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: pthreadpool * URL : https://github.com/Maratyszcza/pthreadpool * License : BSD-2 Programming Lang: C++/C Description : pthread-based thread pool for C/C++ pytorch dependency debian deep learning team
Bug#958654: ITP: psimd -- Portable 128-bit SIMD intrinsics
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: psimd * URL : https://github.com/Maratyszcza/psimd * License : MIT Programming Lang: C++ Description : Portable 128-bit SIMD intrinsics pytorch dependency debian deep learning team
Bug#958653: ITP: ideep -- Chainer module providing numpy like API and DNN acceleration using MKL-DNN
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: ideep * URL : https://github.com/intel/ideep * License : MIT Programming Lang: C++ Description : Chainer module providing numpy like API and DNN acceleration using MKL-DNN pytorch(caffe2) dependency debian deep learning team
Bug#958652: ITP: foxi -- ONNXIFI with Facebook Extension
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: foxi * URL : https://github.com/houseroad/foxi/ * License : MIT Programming Lang: C Description : ONNXIFI with Facebook Extension pytorch(caffe2) dependency debian deep learning team
Bug#958651: ITP: fxdiv -- C99/C++ header-only library for division via fixed-point multiplication by inverse
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: fxdiv * URL : https://github.com/Maratyszcza/FXDiv * License : MIT Programming Lang: C++ Description : C99/C++ header-only library for division via fixed-point multiplication by inverse pytorch dependency debian deep learning team
Bug#958650: ITP: fp16 -- Conversion to/from half-precision floating point formats
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: fp16 * URL : https://github.com/Maratyszcza/FP16 * License : MIT Programming Lang: C++ Description : Conversion to/from half-precision floating point formats pytorch dependency debian deep learning team
Bug#958517: ITP: rabit -- Reliable Allreduce and Broadcast Interface for distributed machine learning
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: rabit * URL : https://github.com/dmlc/rabit/ * License : BSD-3 Programming Lang: C++ Description : Reliable Allreduce and Broadcast Interface for distributed machine learning xgboost dependency maintained under deep learning team
Bug#958516: ITP: asmjit -- Complete x86/x64 JIT and AOT Assembler for C++.
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: asmjit * URL : https://github.com/asmjit/asmjit * License : Zlib license Programming Lang: X++ Description : Complete x86/x64 JIT and AOT Assembler for C++. pytorch dependency. will be maintained under deep learning team.
Bug#958294: ITP: dmlc-core -- A common bricks library for building scalable and portable distributed machine learning.
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: dmlc-core * URL : https://github.com/dmlc/dmlc-core/ * License : Apache-2.0 Programming Lang: C++ Description : A common bricks library for building scalable and portable distributed machine learning. under deep learning team.
Bug#958072: ITP: xgboost -- Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: xgboost * URL : https://github.com/dmlc/xgboost * License : Apache-2 Programming Lang: C++, Pyhton Description : Scalable, Portable and Distributed Gradient Boosting (GBDT, GBRT or GBM) Library Useful for the data science community. Will be maintained under the debian deeplearning team.
Bug#956264: ITP: onedal -- oneAPI Data Analytics Library (oneDAL)
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: onedal * URL : https://github.com/oneapi-src/oneDAL * License : Apache-2 Programming Lang: C++, SYCL Description : oneAPI Data Analytics Library (oneDAL) This is possibly previously known as intel DAAL, a proprietary machine learning library with good performance. I'm not quite sure whether intel changed the proprietary license of DAAL to Apache-2, or created a new implementation.
Bug#956263: ITP: onemkl -- oneAPI Math Kernel Library (oneMKL) Interfaces
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: onemkl * URL : https://github.com/oneapi-src/oneMKL * License : Apache-2 Programming Lang: C++, OpenCL (maybe SYCL) Description : oneAPI Math Kernel Library (oneMKL) Interfaces It looks like intel is starting a brand new MKL implementation to incorprate OpenCL (GPU acceleration). I don't know how Intel will deal with its intel-mkl (proprietary) and oneMKL (apache-2) which provide quite similar functionality. So I'm keeping an eye on this.
Re: Overinterpretation of DFSG? QR code for receiving donation is non-free???
Hi, The upstream author responded quickly: https://github.com/pymumu/smartdns/issues/452#issuecomment-606079773 (in Chinese) The center pictures is actually the user avatars. The avatar in alipay_donate.jpg comes from Nentendo game. The avatar in wechat_donate.jpg is a photo taken by the upstream author. On the other hand, I've suggested the author to remove the Nentendo game avatar: https://github.com/pymumu/smartdns/issues/452#issuecomment-607121122 (in Chinese) On Mon, Mar 30, 2020 at 03:01:36PM +0000, Mo Zhou wrote: > Hi Johannes, > > I opened an issue at upstream: > https://github.com/pymumu/smartdns/issues/452 > (Note, written in Chinese) > > Let's wait to see the answer. > > On Mon, Mar 30, 2020 at 10:49:34AM +0200, Johannes Schauer wrote: > > Hi, > > > > Quoting Paul Wise (2020-03-30 10:20:08) > > > On Mon, Mar 30, 2020 at 7:53 AM Shengjing Zhu wrote: > > > > 1. There's no info loss if you convert from one to another. > > > You definitely lose the (presumably non-free) television/game characters > > > when > > > converting from the original QR codes to plain text. > > > > I think this point is important: are these pictures in the center DFSG free > > or > > not? > > > > lumin, could you answer that question? > > > > Thanks! > > > > cheers, josch
Re: Overinterpretation of DFSG? QR code for receiving donation is non-free???
On Wed, Apr 01, 2020 at 10:37:07AM +0200, Gard Spreemann wrote: > Or, another example that I can imagine plausibly arising in practice: > suppose a terrabyte of raw data was collected from a scientific > experiment or simulation in order to produce (among other things) a plot > in the form of a 100 KB image that it is useful to distribute in > documentation that goes along with some DFSG-free code. Clearly the > "preferred form of modification" is the raw data, together with the code > that processed it, but it seems unpractical to expect the maintainer to > go through a laborious process that perhaps even requires highly > specialized expertise in order to distribute the raw data and reassemble > the plot from it. In this case I agree that distributing the resulting image as is, is the sensible thing to do. Another thing came up in my mind, quite interesting, is that if the resulting product is a pre-trained neural network, I get completely reversed conclusion ... Further, some neural networks are trained from the wikipedia dump (CC-licensed). Are we uploading wikipedia dumps to the archive when our users need to use these models? What's the essential difference between a jpg picture and a pretrained neural network then? They are both multi-dimensional numerical arrays from an abstract perspective, but a normal human can understand and modify the picture pixels, while not being able to understand or modify the network prarameters.
Re: Overinterpretation of DFSG? QR code for receiving donation is non-free???
Hi Johannes, I opened an issue at upstream: https://github.com/pymumu/smartdns/issues/452 (Note, written in Chinese) Let's wait to see the answer. On Mon, Mar 30, 2020 at 10:49:34AM +0200, Johannes Schauer wrote: > Hi, > > Quoting Paul Wise (2020-03-30 10:20:08) > > On Mon, Mar 30, 2020 at 7:53 AM Shengjing Zhu wrote: > > > 1. There's no info loss if you convert from one to another. > > You definitely lose the (presumably non-free) television/game characters > > when > > converting from the original QR codes to plain text. > > I think this point is important: are these pictures in the center DFSG free or > not? > > lumin, could you answer that question? > > Thanks! > > cheers, josch
Re: Overinterpretation of DFSG? QR code for receiving donation is non-free???
Hi, Another public acknowledgement I should make is that I'm no longer an FTP Trainee ... So my name in the member list [1] is no longer valid. In the past indeed I have discussed about my shallow experience of working as a Trainee, and discussed about how some portions of the NEW queue workflow can be improved. According to the private messages I have received, I started to doubt whether I should have done that. So ... sorry again if that introduced inconvenience to you guys. This is not related to anything about transparency. This is just that I violated the current rules of the team. And I accept the consequence of being expelled from the team and the damage to trust. Well, sometimes I'm just so blind to see only one thing and get braindead. [1] https://ftp-master.debian.org/ On Mon, Mar 30, 2020 at 02:28:19PM +0000, Mo Zhou wrote: > Hi fellow devs, > > I acknowlege that I sometimes do things in inappropriate ways > unintentionally, and I accept the consequences of my fault. And this > time I did something unprofessional, leaking messages from -private > without asking for permission first. > > I was wrong. Sorry for that. > > Mo. > > On Mon, Mar 30, 2020 at 05:54:23AM +, Mo Zhou wrote: > > Hi fellow devs, > > > > I think sometimes the DFSG has been over-interpreted. Here I'm talking about > > the recent REJECTion of src:smartdns from our NEW queue, where QR code > > pictures > > used for donation have been deemed DFSG non-free [1]. I'm not satisfied with > > the explanation, and I think there is over-interpretation on DFSG. > > > > I poked ftp-master about this problem: > > > >spwhitton: I'm quite confused about REJECTION of src:smartdns. > > Why can > > the QR code pictures for software author to receive donations be > > DFSG-nonfree? > > > > And I got the following explanations: > > > >lumin: IIRC that was not the only reason for REJECT. > > Otherwise I > > would have PRODded. > > > >lumin: An image of a QR code wouldn't be the preferred form of > > modification. They are usually generated from something. If the file it > > was > > generated from isn't present and the tool to generate it isn't in Debian, > > then > > it can't be shipped. Requiring preferred form of modification is one area > > where Debian is often stricter than licenses due to DFSG. > > > > The pictures we're talking about are: > > > > * > > https://salsa.debian.org/debian/smartdns/-/blob/master/doc/alipay_donate.jpg > > * > > https://salsa.debian.org/debian/smartdns/-/blob/master/doc/wechat_donate.jpg > > > > "alipay" and "wechat" are the top-2 domination payment platforms in > > Chinese > > market. And the two QR code pictures are generated from the corresponding > > APPs by the upstream author. The whole software project is licensed under > > GPL-3 and the QR codes are used for receiving donations. > > > > Why are they non-free? > > > > Treating this files as non-free could lead to further problems. > > > > 1. If I stripped the donation codes from the source. > > I believe such behaviour is **unethical**. > > > > 2. If I decoded the QR code and replaced them with the underlying URLs. > > There is no Chinese user who pay through URL instead of QR code. > > > > 3. If I stripped the donation codes but re-generated them during the > > package > > build process. > > "Oh damn, this QR code does not look like the original one and the > > hashsum > > mismatches. Has the Debian developer forged the QR code to be evil?" > > I mean there will be doubt if the distributed QR code is not > > byte-to-byte > > equivalent to the one distributed by upstream author. > > > > Is a QR code for donation really DFSG non-free? Is DFSG over-interpreted in > > this case? How should package maintainers deal with QR codes ethically? > > > > [1] The package has been REJECT'ed for two reasons: > > 1. "doc/*_donate.jpg are probably not DFSG-free" > > 2. Missing copyright information for > > "package/luci-compat/tool/po2lmo/src/*" > > There is no problem with the second point. This mail only talks about > > the first point. signature.asc Description: PGP signature
Re: Overinterpretation of DFSG? QR code for receiving donation is non-free???
Hi fellow devs, I acknowlege that I sometimes do things in inappropriate ways unintentionally, and I accept the consequences of my fault. And this time I did something unprofessional, leaking messages from -private without asking for permission first. I was wrong. Sorry for that. Mo. On Mon, Mar 30, 2020 at 05:54:23AM +, Mo Zhou wrote: > Hi fellow devs, > > I think sometimes the DFSG has been over-interpreted. Here I'm talking about > the recent REJECTion of src:smartdns from our NEW queue, where QR code > pictures > used for donation have been deemed DFSG non-free [1]. I'm not satisfied with > the explanation, and I think there is over-interpretation on DFSG. > > I poked ftp-master about this problem: > >spwhitton: I'm quite confused about REJECTION of src:smartdns. Why > can > the QR code pictures for software author to receive donations be > DFSG-nonfree? > > And I got the following explanations: > >lumin: IIRC that was not the only reason for REJECT. Otherwise > I > would have PRODded. > >lumin: An image of a QR code wouldn't be the preferred form of > modification. They are usually generated from something. If the file it > was > generated from isn't present and the tool to generate it isn't in Debian, > then > it can't be shipped. Requiring preferred form of modification is one area > where Debian is often stricter than licenses due to DFSG. > > The pictures we're talking about are: > > * > https://salsa.debian.org/debian/smartdns/-/blob/master/doc/alipay_donate.jpg > * > https://salsa.debian.org/debian/smartdns/-/blob/master/doc/wechat_donate.jpg > > "alipay" and "wechat" are the top-2 domination payment platforms in Chinese > market. And the two QR code pictures are generated from the corresponding > APPs by the upstream author. The whole software project is licensed under > GPL-3 and the QR codes are used for receiving donations. > > Why are they non-free? > > Treating this files as non-free could lead to further problems. > > 1. If I stripped the donation codes from the source. > I believe such behaviour is **unethical**. > > 2. If I decoded the QR code and replaced them with the underlying URLs. > There is no Chinese user who pay through URL instead of QR code. > > 3. If I stripped the donation codes but re-generated them during the package > build process. > "Oh damn, this QR code does not look like the original one and the > hashsum > mismatches. Has the Debian developer forged the QR code to be evil?" > I mean there will be doubt if the distributed QR code is not byte-to-byte > equivalent to the one distributed by upstream author. > > Is a QR code for donation really DFSG non-free? Is DFSG over-interpreted in > this case? How should package maintainers deal with QR codes ethically? > > [1] The package has been REJECT'ed for two reasons: > 1. "doc/*_donate.jpg are probably not DFSG-free" > 2. Missing copyright information for > "package/luci-compat/tool/po2lmo/src/*" > There is no problem with the second point. This mail only talks about the > first point. signature.asc Description: PGP signature
Overinterpretation of DFSG? QR code for receiving donation is non-free???
Hi fellow devs, I think sometimes the DFSG has been over-interpreted. Here I'm talking about the recent REJECTion of src:smartdns from our NEW queue, where QR code pictures used for donation have been deemed DFSG non-free [1]. I'm not satisfied with the explanation, and I think there is over-interpretation on DFSG. I poked ftp-master about this problem: spwhitton: I'm quite confused about REJECTION of src:smartdns. Why can the QR code pictures for software author to receive donations be DFSG-nonfree? And I got the following explanations: lumin: IIRC that was not the only reason for REJECT. Otherwise I would have PRODded. lumin: An image of a QR code wouldn't be the preferred form of modification. They are usually generated from something. If the file it was generated from isn't present and the tool to generate it isn't in Debian, then it can't be shipped. Requiring preferred form of modification is one area where Debian is often stricter than licenses due to DFSG. The pictures we're talking about are: * https://salsa.debian.org/debian/smartdns/-/blob/master/doc/alipay_donate.jpg * https://salsa.debian.org/debian/smartdns/-/blob/master/doc/wechat_donate.jpg "alipay" and "wechat" are the top-2 domination payment platforms in Chinese market. And the two QR code pictures are generated from the corresponding APPs by the upstream author. The whole software project is licensed under GPL-3 and the QR codes are used for receiving donations. Why are they non-free? Treating this files as non-free could lead to further problems. 1. If I stripped the donation codes from the source. I believe such behaviour is **unethical**. 2. If I decoded the QR code and replaced them with the underlying URLs. There is no Chinese user who pay through URL instead of QR code. 3. If I stripped the donation codes but re-generated them during the package build process. "Oh damn, this QR code does not look like the original one and the hashsum mismatches. Has the Debian developer forged the QR code to be evil?" I mean there will be doubt if the distributed QR code is not byte-to-byte equivalent to the one distributed by upstream author. Is a QR code for donation really DFSG non-free? Is DFSG over-interpreted in this case? How should package maintainers deal with QR codes ethically? [1] The package has been REJECT'ed for two reasons: 1. "doc/*_donate.jpg are probably not DFSG-free" 2. Missing copyright information for "package/luci-compat/tool/po2lmo/src/*" There is no problem with the second point. This mail only talks about the first point. signature.asc Description: PGP signature
Re: FTP-Master Review Process
Hi fellow devs,
Just appending some of my side notes:
Michael's ftpmaster-proposal shares some common motivations with my
previous mail ("Intermediate representation ... license review"), and we
basically reached in an agreement on the problems existing in our
FTP-master working (NEW) process. The difference between our proposals
is that mine focuses on a small, specific, and dedicated problem, while
Michael's concerns problems in a larger-scale perspective -- a superset
of mine. Or say bottom-up v.s. top-down.
I'm still asynchronously working on my tool for helping NEW queue reviewing
process, and it may solve a portion of the issues pointed out by the proposal.
But note that my action on this will be slow due to my $college stuff.
When have enough time, I will document the core thoughts and
specifications of my work-in-progress project and post them here.
Simple demo code will come after that.
On Sun, Mar 29, 2020 at 05:37:01AM -0500, Michael Lustfield wrote:
> Hello fellow masochists,
> ... err, I mean Debian Developers,
>
> There has been a lot of talk lately about the FTP-Masters team and how some
> issues should be fixed. Back in December, I alluded to a proposal that I was
> working on. I hoped to get a little further into writing source prior to this
> point, but life happens and now seems to be a much more appropriate time than
> later.
>
>
> In my "proposal" [1], I outline the concerns that have been noted (from
> mailing
> lists, IRC, and private discussions) and then provide a few potential
> solutions.
>
> Although some source code exists, this is a discussion for later. I want to
> take
> a very systematic and structured approach to this.
>
> To outline...
> 1. Identify a problem
> 2. Evaluate the cause
> 3. Consider solutions
> 4. Evaluate the solution
> - Does it solve the problem?
> - Does it introduce new problems?
> - What will it take to implement?
> - Would any developer be willing to work on it?
> - Will it be maintainable in the long-term?
> - etc.
> 5. Build some prototypes
> 6. Re-evaluate the solution (copy #4)
> 7. Finish design requirements
> 8. Start building
> 9. Lotsa testing
> 10. Start discussing implementation
>
> Note... I am using the term proposal because very little of what I put forth
> is
> in any way a design document. It's just many thoughts tossed at a page that
> now
> require critical review. (critical -- be picky, not rude)
>
> Once the discussion is over (or degrades), I will begin writing a design
> document. After complete, I will ask for additional comment and interested
> developers.
>
>
> Side note-
> An important road block to be aware of is that implementing almost any of the
> proposed solutions will require significant (time & effort) changes to the
> main
> archive server. It will also likely require some legal counsel. I'll omit the
> gritty details since this is mostly just FYI, but it's also worth considering
> team size and availability (see: $current_issue) when it comes to
> implementation.
>
>
> [1] https://salsa.debian.org/mtecknology/ftpmaster-proposal
>
> --
> Michael Lustfield
Re: trimming changelogs
Hi Adam,
Thank you for putting this forward, but actually I'm personally
object to trimming changlogs from a normal[1] system. A complete
changelog helps me quickly understand what happened to the
component of the package. Once trimmed, such changelog get crippled. Why
not directly keep only the latest dch section? People who need the full
log can look up the packaging repository.
That said, I agree that we move a step further considering a slim base
system in a more constructive way. Spontaneously, I'd raise such a
question: Is the changelogs the only files to blame for the waste of
precious space? Some packages contains lots of unnecessary HTML/PDF/.*
documentations/resource/assets which are very likely useless on those
low-storage systems. Who is going to read long documentations on a weak
embedded device?
Here is my bold idea: we create a (new) special mode oriented for
low-storage systems. For important packages (that exist in stage{3,4}),
maintainers mark those unnecessary files as "trimmable". On
installation, the dpkg automatically ignores the "trimmable" files
once the "low-storage" mode has been toggled. For differnent "trimmable"
files we can write different trimming helpers (such as the one for
changelogs).
^^^ In this way we may be able to solve all system storage headaches for
people working on embedded device.
If we decide to invest our energy in trimming the base system, we should
think about the "trimming stage3" problem in a more systematical way.
[1] where 10MiB less available space is never a problem
On Fri, Mar 20, 2020 at 12:50:29AM +0100, Adam Borowski wrote:
> Hi!
> In the rush for cutting away small bits of minbase, it looks like we forgot
> a big pile of junk: /usr/share/doc/
>
> On strict minbase (rather than prio:important which really matters), the
> docs take 11MB. And of that, 8MB are files named changelog.* -- which
> fails to include eg. bash's:
> 112K -rw-r--r-- 1 root root 111K Jan 2 2019 CHANGES.gz
> 8.0K -rw-r--r-- 1 root root 8.0K May 29 2018 COMPAT.gz
> 4.0K -rw-r--r-- 1 root root 2.9K Feb 17 1999 INTRO.gz
> 32K -rw-r--r-- 1 root root 30K Nov 13 2018 NEWS.gz
> (INTRO.gz is small, but 1999 advertising is of little use)
>
> Of files named changelog.*, top offenders are:
> 880997 dpkg:changelog.gz
> 381250 gpgv:changelog.gz
> 289255 libgnutls30:changelog.gz
> 223009 ncurses-bin:changelog.gz
> 223009 ncurses-base:changelog.gz
> 223009 libtinfo6:changelog.gz
> 210621 libc6:changelog.Debian.gz
> 210621 libc-bin:changelog.Debian.gz
> 202841 dpkg:changelog.Debian.gz
> 193825 coreutils:changelog.gz
> 177117 gcc-9-base:changelog.Debian.gz
> 176608 gcc-10-base:changelog.Debian.gz
> 164010 findutils:changelog.gz
> 147656 tar:changelog.gz
> 145889 libapt-pkg6.0:changelog.gz
> 145889 apt:changelog.gz
> 145271 passwd:changelog.gz
> 145271 login:changelog.gz
> 142058 grep:changelog.gz
> 131424 libp11-kit0:changelog.gz
> 123812 libgcrypt20:changelog.gz
> 113533 bash:changelog.gz
> 103492 libnettle7:changelog.gz
> 100216 libpcre3:changelog.gz
> 93638 libudev1:changelog.Debian.gz
> 93638 libsystemd0:changelog.Debian.gz
> 63709 perl-base:changelog.Debian.gz
> 63670 logsave:changelog.Debian.gz
> 63670 libss2:changelog.Debian.gz
> 63670 libext2fs2:changelog.Debian.gz
> 63670 libcom-err2:changelog.Debian.gz
> 63670 e2fsprogs:changelog.Debian.gz
> 63284 tar:changelog.1.gz
>
> Seems like a tempting area to trim...
>
>
> Prior art:
> sysvinit does:
> sed -i -ne '/sysvinit (2.93-8)/q' -e p \
> $(rctmp)$(doc)/sysv-rc/changelog.Debian
> (I've just bumped the cut-off from 2.86.ds1-47, in 2007)
> which also differs per binary package.
>
> Ubuntu keep only 10 last entries, for _all_ packages.
>
>
> I consider 10 entries to be too little for a fast moving package ("upload
> early, upload often"), but a release-based ("since oldstable"), time-based
> ("3 years ago") or size-based ("X 4096 filesystem pages after gzipping")
> cut-off would work well.
>
> On the other hand, changelogs are valuable. Unlike some folks on IRC
> I wouldn't want to tightly trim all packages. Unlike minbase or
> prio:important, your average 5GB install doesn't care about a few megs
> here and there. Thus: do we want to trim manually or globally?
>
> A global trim would require a lot less work. A manual trim would allow
> managing packages: dpkg is everywhere, dpkg-dev is not. libsystemd0 is
> essential, systemd doesn't belong in containers. gcc-9-base is included
> on tiny installs, gcc-9 on dev boxes and buildds with plenty of space.
> Plus, manual trimming would also allow axing old upstream cruft.
>
> Thoughts, folks?
>
>
> Meow!
> --
> ⢀⣴⠾⠻⢶⣦⠀
> ⣾⠁⢠⠒⠀⣿⡁ in the beginning was the boot and root floppies and they were good.
> ⢿⡄⠘⠷⠚⠋⠀ -- on #linux-sunxi
> ⠈⠳⣄
>
Re: using salsa as a package triage system? "channels"?
Hi Steffen, I still don't quite understand the proposed idea. Is it something like the Debian version of Gentoo overlay? My suggestion is that you should make this proposal more specific, e.g. by shedding some light on the way of implementing it. Unlike the other distros, as you must have noticed, the human cost per package for debian is significantly higher than that for other distros -- an Archlinux developer may update thousands of packages a day, but for me (Debian) 10 packages per day is already lots of work. That means third-party package channels will inherit the same cost per package, even if such cost can be reduced by doing something unelegant. In this direction I proposed DUPR in the past and provided an unelegant demonstration implementation. We also have some similar historical proposals, such as the Debian PPA as you said, and the bikeshed (I don't know who named it but please rename it for good wish). On Sun, Mar 15, 2020 at 04:22:47PM +0100, Steffen Möller wrote: > masters maybe we have to reconsider - after all, the FTP masters only > see a fraction of what we really need in the distribution to solve > today's biological challenges. Sometimes you can tell FTP masters the important things via the "Comments:" field in d/copyright as long as appropriate. FTP masters and trainees won't skip this field. > want to follow. This pragmatic stance has been taken by conda and while > our scrutiny is much accepted and praised in the scientific community, > our packaging falls behind in both scientific coverage and adoption rate. Traditional packaging is lagging behind, or cannot even satisfy the user demand. But that's exactly why business companies like anaconda exists. As discussed before, I think we such a traditional distribution cannot easily solve the problems that anaconda is trying to solve. > Conda folks suggest that Debian should concentrate > on the basic OS. And they have a point, especially since Conda gets > increasingly close to what Debian is offering, e.g. with respect to CI. I fully agree with conda folks that Debian should concentrate on the basic OS, providing a solid base system for whoever the end user is. Although one could have much fun working on low-popcon packages, but the fact that "there won't be too many end users" is already demotivating enough from time to time. > We cannot immediately adopt how conda is doing it since we would lose > what makes us special. But we can learn and could imho fairly easily > rewire Debian's package flow to make being an FTP master fun again. And > at the same time speed up the periphery. Suggestions: > > * the periphery gets their own repositories - happily also called > "channel" in analogy to conda > * maintainer decides on what packages are auto-update-able within the > channel by routine-updates (referencing a script by ) > * channel-community upvotes the integration of packages with Debian > proper as a suggestion for FTPmasters > * popcon gives feedback on the adoption of packages > * FTPmasters pick new packages from the periphery channels as they see fit This part looks obscure to me. Could you provide more specific details? > I am not sure about the granularity of these channels. Maybe every blend > should have one or two channels. Obvious (to me) candidates: > > * med > * astro > * science > * r > * machine learning > * This indeed sounds like Gentoo overlay. I like gentoo overlay. I also fancy gentoo features that Debian doesn't have. > We had discussed PPAs in the past - this suggestion is admittedly > similar. The interaction with salsa and ftpmasters is what renders it > most different, I tend to think. It is not an issue only of Debian Med - > the packages are too much interconnected, I think. The packages that > have most reverse dependencies to packages of many channels are the ones > that the FTPmasters may want to prioritize. Such priority value can be calculated, say, by counting the number of reverse dependencies. Someone interested in this could try to implement this feature in dak and the FTP team will see this hint and tell the priorities.
Re: FTP Team -- call for volunteers
Thank you Sean. I believe what you do is significantly meaningful for our FTP team and the whole project. The recruitment cycle of FTP team is "notoriously" slow, and any effort put on it is much appreciated! On Sat, Mar 14, 2020 at 01:37:59PM -0700, Sean Whitton wrote: > Hello everyone, > > The FTP Team is seeking volunteers to train to become FTP Assistants. > > This is almost entirely a matter of learning how to process the NEW > queue -- learning how to process requests for removals, override changes > etc. is typically something one does *after* becoming an Assistant. > Contributing code to DAK is not a requirement for team members and > contributing code to DAK does not require team membership. > > I have been tasked with supervising both trainees and the training > process, but you can expect to receive feedback from all of the team > members who work NEW. > > Requirements: > > - must be a DD > > - can work with all existing team members > > - comfortable with browsing source packages over SSH using text tools > (packages in NEW must not be downloaded from ftp-master.d.o to your > local machine) > > - willing to wade through legal texts and endure any resultant > headaches > > - cope well with flames in response to your decisions > > - should be an experienced packager. In addition to > licensing/copyright issues, NEW review attempts to catch bugs where > the package entering the archive would make life difficult for > contributors > > - after training, comfortable with being on the other end of the > ftpmaster@ alias, which receives a huge volume of > not-always-pleasant messages daily. > > Some other info in these old calls for volunteers: > > - https://lists.debian.org/debian-devel-announce/2010/03/msg3.html > - https://lists.debian.org/debian-devel/2017/08/msg00408.html > > And in our docs: > > - https://ftp-master.debian.org/REJECT-FAQ.html > - https://salsa.debian.org/ftp-team/manpages > > Write to if interested. > > For the FTP Team, > > -- > Sean Whitton
Bug#951307: ITP: smartdns -- local DNS server to obtain the fastest IP for the best experience
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: smartdns * URL : https://github.com/pymumu/smartdns/releases * License : GPL-3+ Programming Lang: C Description : local DNS server to obtain the fastest IP for the best experience The package is already usable here: (I've not done the the code check and license check yet) https://salsa.debian.org/debian/smartdns I'll upload to to NEW after finishing the remaining checks and my local assessment.
Bug#951205: ITP: dlpack -- Open In Memory Tensor Structure and operator interface for deep learning
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: dlpack Version : git head * URL : https://github.com/dmlc/dlpack * License : apache-2.0 Programming Lang: C Description : Open In Memory Tensor Structure and operator interface for deep learning mxnet dependency
Re: call for ftpmaster's transparency
On Sun, Feb 09, 2020 at 07:01:05PM -0700, Sean Whitton wrote:
> One key problem with the current workflow is that it makes it very
> difficult to avoid reviewing identical files more than once. That would
> be a big improvement.
(I was just talking with Michael about this several minutes ago.)
Just leaking a part of my WIP work.
My core data structure looks like this
{path: [hash, stamp, username, status, annotation]}
The "hash " field is a salted hash, calculated like this
hash(data=read(path), salt=read(neighbor_license()))
This data structure is a fine-grained (per-path level) "accept/reject"
record. Each path is a node. The "status" of a tree can be
automatically computed form its decendant nodes.
When a package enters NEW again, files with matching hashes will
automatically reuse the last status assigned by human user, where status
= either "accept" or "reject".
There are still many other aspects from which I can reduce time
consumption for human and improve efficiency.
Re: call for ftpmaster's transparency
Hi Niels, On Sun, Feb 09, 2020 at 11:22:46AM +0100, Niels Thykier wrote: > For the parts involving tooling, are there bugs/salsa issues describing > the issue so a "non-FTP-team"-member can take a stab at fixing them? First of all the major problem we are talking about, that the reviewing process being so slow, is not well defined. Because what this is going to optimize is not the machine code, but the human's working procedure. So it's not easy for people to open a bug and describe the definite problem they have found. For instance, bugs saying "hi dak devs, I found the tool not friendly enough for human efficiency, please fix it" are very likely terrible bug reports. Something new needs to be invented (and eventually incorporated into somewhere like dak). I happen to have a bottom-up idea that is still a work-in-progress[1]. Albeit I believe what I'm doing can greatly facilitate my NEW reviewing process (as a trainee), I still suffer from the lack of time and energy to push the draft forward. I had some further private discussion with Michael and Dmitry. People's opinions on the solution differ. So I speculate that the community will have to come up with some more concrete ideas and experiment them a little bit to settle the whole NEW-SO-SLOW issue. [1] see my old post: "Idea: intermediate license review"
Re: GSoC projects about SUSI.AI
On Thu, Feb 06, 2020 at 01:10:36PM +0900, Norbert Preining wrote: > What do you mean "packages involved". The core ML related package is by > now not completely free (snowboy) thus we have prepared a project that > also tries to incorporate Mycroft or some other OSS hotword module > which allows for user-provided training. "Packages involved" means the packages that you intend to upload to our archive. The freeness of plain text files (e.g. code) is easy to judge. When you have doubt whether to upload a binary blob (pretrained model) to main or contrib, feel free to ask me publically. For example, these looks like typical questionable models: https://github.com/Kitt-AI/snowboy/tree/master/resources/models Without corresponding training data (FOSS-licensed) being publically available, these models should not be considered free regardless of the license of the binary blobs. > By now we use snowboy with a model trained using recordings we have > provided. In the future, as said, we want something that is completely > DFSG compliant, this is one of the aims. Great. > What are the definitions of DFSG-compliant for models? I guess that is > written in [1] I interpret DFSG-compliant model as the "Free Model" defined in [1]. I need to update the document when I got enough free time to do so. > I agree, this is a complicated area and I think we can make good > progress with proper compliance with DFSG during GSOC. Good luck.
Re: call for ftpmaster's transparency
Spiritually I really would like to see a transparent workflow of the FTP team. If it were a couple years ago I may stand in the same position as you. But now I'd kindly invite you to review the FTP team workflow (I joined the FTP team in order to review it), review the functionalities of dak (at least we trainees depend on it, which means what dak can do directly decides whether the process can be made transparent), and think of the possible ways we can make things better. That could provide us a more constructive start. Isn't it? Sam, IMHO DPL does not have to lead every important topics in this community, as that would be exhausting. On Thu, Feb 06, 2020 at 10:32:42AM +1100, Dmitry Smirnov wrote: > IMHO it is disturbing that one of the most essential processes in Debian > -- acceptance of new and modified packages -- operates almost in secrecy. > > Unlike most Debian teams, ftp-masters communicate in private mail list. > I understand why security team might need to operate without full public > disclosure but I see no reason for ftp-masters to avoid transparency. > Wouldn't it be easier to understand what to expect if everyone could see > how team operates? > > To make matters worse ftp-masters rarely leave their comments in ITP > issues. As I've recently learned that have profound effect on processing of > new packages. > > One of my packages spent a year in the NEW queue at some point raising to > position number 4. Apparently before release of Buster (2019-07-06) member > of ftp-masters team left an internal (invisible to the public) comment on > my package that was not communicated to me until 7 months later when my > package was rejected based on that comment. The comment could have been > addressed without delay if it was left on the corresponding ITP issue where > it belong. > > A precious time was lost but more importantly one can see that current > process requires an extra effort to communicate with maintainers -- a > something that would not be necessary if ftp-masters use the official > channel that exist specifically to discuss introduction of new packages -- > ITP bug reports. > > I'd like Debian project leader to engage in the matter of improving > transparency of ftp-masters team operations and procedures. > > As very minimum I recommend to change current ftp-master procedures to use > ITP bugs instead of internal comments whenever possible, for the sake of > transparency and to optimise communication. > > I want to encourage a public discussion regarding opening of the ftp-master > mail list to the public. Currently reasons for unjustified secrecy of ftp- > master processes is not explained... > > https://wiki.debian.org/Teams/FTPMaster > https://wiki.debian.org/NewQueue > > -- > All the best, > Dmitry Smirnov. > > --- > > Honesty is a gift we can give to others. It is also a source of power and > an engine of simplicity. Knowing that we will attempt to tell the truth, > whatever the circumstances, leaves us with little to prepare for. We can > simply be ourselves. > -- Sam Harris
GSoC projects about SUSI.AI
Hi Norbert, As you know, I cannot help to care about every topic about Debian + AI. By chance I saw some interesting pending GSoC projects: https://wiki.debian.org/SummerOfCode2020/Projects Packaging SUSI.AI and dependencies for Debian Develop Debian AI Voice Assistant Blend For good wish, could you please make a list of packages involved in these projects, so I can inspect them and make sure that the ML-Policy[1] (mostly about software freedom) won't get in your way? In particular, I'll confirm the following questions: 1. does it use any kind of pre-trained machine learning model? 2. is the pre-trained model free and DFSG-compliant? Thanks in advance. [1] ML-Policy is out of date ... well I was too busy ...
Re: opentmpfiles & opensysusers, and its use in the Debian policy
Hi Thomas, Thank you for working on these! On Thu, Jan 02, 2020 at 02:14:34PM +0100, Thomas Goirand wrote: > As I wrote, no need to complain, but act. > > https://salsa.debian.org/debian/opentmpfiles > https://salsa.debian.org/debian/opensysusers > > Both are now in the NEW queue, and both need some kind of init.d sysv-rc > script. Please anyone, contribute that init script. I quickly scanned the etc files in my Gentoo chroot, and found some openrc scripts for opentmpfiles. I volunteer to translate that one into the form we can use, but currently I don't acquire the lock due to my overflowing todo list. [1] For opensysusers I suspect that Gentoo didn't package it as $ emerge --search opensysusers shows nothing. Maybe we can also borrow something from the non-systemd variant of Archlinux. [1] Does anyone know if systemd-nspawn can (theoretically) boot a non-systemd stage3 image? I tried to boot a gentoo+openrc image, but it ended up stuck somewhere. If this is doable, it will make my init script debugging and testing process much much less painfull.
Re: Bug#947817: ITP: kaldi -- Kaldi Speech Recognition Toolkit
On Tue, Dec 31, 2019 at 08:45:26AM +, Mo Zhou wrote: > Hi bartm, > > I'm wondering why you would like to close my ITP bug, although I found a > similar existing RFP several minutes ago. Please ignore my question. The BTS dropped all the content following the "stop" command. So I was quite confused when informed by the BTS. Now I see the full message, including the reason. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=947817;msg=7
Re: Bug#947817: ITP: kaldi -- Kaldi Speech Recognition Toolkit
Hi bartm, I'm wondering why you would like to close my ITP bug, although I found a similar existing RFP several minutes ago. Coincidentally, upstream does not encourage linux distro integration, as they believe their toolkit is intended for expert users only. https://github.com/kaldi-asr/kaldi/issues/2946#issuecomment-569865593 That means I would close this ITP even if you didn't do it. On Tue, Dec 31, 2019 at 04:57:47AM +0000, Mo Zhou wrote: > Package: wnpp > Severity: wishlist > Owner: Mo Zhou > > * Package name: kaldi > * URL : https://github.com/kaldi-asr/kaldi > * URL : http://kaldi-asr.org/ > * License : Apache-2 > Programming Lang: C++, python, etc. > Description : Kaldi Speech Recognition Toolkit > > This is a widely used Automatic Speech Recognition (ASR) toolkit. > It provides me more experience for developing ML-Policy. >
Bug#947817: ITP: kaldi -- Kaldi Speech Recognition Toolkit
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: kaldi * URL : https://github.com/kaldi-asr/kaldi * URL : http://kaldi-asr.org/ * License : Apache-2 Programming Lang: C++, python, etc. Description : Kaldi Speech Recognition Toolkit This is a widely used Automatic Speech Recognition (ASR) toolkit. It provides me more experience for developing ML-Policy.
Re: Idea: frontend tool for more efficient license reviewing based on tree-structured IR
On Fri, Dec 27, 2019 at 04:00:33PM -0600, Michael Lustfield wrote: > > ## Motivations > > I've had similar motivations. Since becoming a Trainee, I've found the review > process to be rather painful. I think the slow and klunky tools we use are a > big problem and likely contribute significantly to the burnout this team sees. Good to hear that from another trainee :) I'm preparing some illustrative examples for my idea and some more design details. Will send updates to this thread soon. > I started a similar effort when I first became a trainee. Unfortunately, a lot > of our non-trainees seem to be burned out, which means no reviews, and no > reviews means a lack of confidence in whatever system I'm building. Is it published somewhere? > I can handle basic UI design and know a couple people that are very good at > it. > I'm also able to host some infrastructure for what I have in mind. Can you > share the source you've been working on? I'm currently turning my notes into > something a bit more intelligible. My old source was merely some sort of UI programming excercise.
Re: Idea: frontend tool for more efficient license reviewing based on tree-structured IR
On Fri, Dec 27, 2019 at 04:54:32PM +0100, Jonas Smedegaard wrote: > Long description of debmake claims it "does more than what > licensecheck(1) offers" but I am puzzled what that sentence means - more > polished experience (even if less accurate), perhaps? IIRC it appends the license texts to the generated d/copyright, such as License: Expat > > I personally do `licensecheck -r --deb-machine . > debian/copyright` > > and manually tweak the content. > > Beware that licensecheck by default omits some files, and for the files > it does check by default it inspects only the top of the file (and the > bottom, but only if nothing was found at the top). I'm aware of that. Basically every template generated in this way will be significantly reorganized before uploading. > > ftp-master: possibly manually reviewing with MC + custom plugin > > > > I didn't follow the recommended way. > > What is "the recommended way"? IIRC using midnight-commander (MC) + custom plugin, according to an introductory document written by the ftp team. > Not sure what you mean above. The format supports wildcards but that's > optional - if you want you can write path for each file explicitly, and > it should be simple to write a tool that converts a copyright file with > wildcards to one without wildcards. I'm preparing some illustrative examples and more design details. > copyright file format permits additional fields. If you want to track > files moving around, I suggest adding (e.g. in same script expanding > wildcards as mentioned above) field FileChecksum as defined by SPDX: > https://spdx.github.io/spdx-spec/4-file-information/#44-file-checksum Does not sound like what I wanted. > > * `licensecheck`. Given a file FILE, `licensecheck FILE` produces the > > license > > name. > > I disagree with "the license name" above - it is not that simple: One > file can be covered by multiple licenses - OR'ed or AND'ed or > uncertain-how-they-relate or uncertain-what-they-are or > uncertain-if-none-found-means-none-there or certainly-none-there. This will be fixed in refined design details. You may have noticed that my intent of the original post is introducing some core (and coarse) concepts/ideas. There were some inaccurate details or even discrepancies, but distracting readers by describing too much in such minor detail is not what I meant to do. > And speaking of uncertainty, several steps in the parsing of > human-written comments can contain different kinds of uncertainty, which > for some use-cases of licensecheck is sensible to err on either side. > > Example: A project containing a minxture of BSD-2 and BSD-3 files which > has historically contained BSD-4 files is important to err on the side > of BSD-4 whenever there is doubt, whereas a project written from scratch > in recent times might make sense to err on the side of "they probably > meant BSD-3" unless certain that it is BSD-4. > > ...or some would argue that it is never sensible to err on either side, > but instead whenever a fuzzy parser is uncertain it must flag the > uncertainty and hand it over to human inspection. > > Another example: Copyright statement for the year 198 might in one > context "obviously" a typo for 1998, in another context most likely be > no-a-copyright-statement-at-all, and in a third context better handed > over to human inspection. > > To summarize, I'd say that as minimum it need to provide a license > _expression_ and a certainty expression. > > For license expression, see > https://spdx.github.io/spdx-spec/4-file-information/#45-concluded-license > and > https://spdx.github.io/spdx-spec/appendix-IV-SPDX-license-expressions/ Undoubtedly this will be adopted. > > * `grep` or `ripgrep`. For example, `rg -i copyright FILE` always > >works well. > > Matching purely by keywords should be tracked as an uncertainly. The usage of `grep` outputs is not the same as you thought. Please look forward to my update. > > * "neighbor". For example, given a source file "F/I/L/E" without any > > copyright & license info, looking for F/I/L/LICENSE, F/I/LICENSE, > > ..., etc like git does for the ".git" directory will help. > > Looking outwards will not help to find _file_ license, only to find > _package_ license (by considering some subset of the Debian package a > virtual "package" in the SPDX terms). Unless you also parse that > LICENSE file to identify which exact files it covers. > > Also, expanding outwards should be tracked as an uncertainty. Ditto. > > The formated+filtered output of any combination of these backends can > > be attached to the corresponding IR. > > > > In contrast, a "frontend" tool is also needed for dealing with such IR > > in a higher level. My imagined "frontend" tool is a `ranger`-like file > > browser with specific designs. > > > > * the user can choose what backend(s) to use. If none is chosen, the > >frontend tool fall
Idea: frontend tool for more efficient license reviewing based on tree-structured IR
Hi fellow devs, I created an amount of NEW packages as a DD, and reviewed an amount of NEW packages in the NEW queue as FTP trainee. Both of the two kinds of work involves an important part -- sometimes annoying -- license checking. People keeps complaining about it, and recently there were some related discussions[1][2] on -project, talking about possible ways to improve -- some in the workflow aspect, the other in the tooling aspect. In this mail I have an idea about tooling. This is a long mail. I've alreay organized it in a structured format so you can apply more fast reading tricks. The problem we are trying to solve -- Given a arbitrary source tree, we shall examine the copyright & license information for each file node, make sure each node complies with DFSG, and make an overall assessment to the whole tree: ACCEPT/REJECT. Subsequently, the tree will be flattened (the tree structure being removed) and written into debian/copyright in machine-readable format. Note that, automatically parsing a machine-UNreadable debian/copyright requires a delicate recurrent neural network. That machine-UNreadable case is too complex, so let's ignore it for now. Existing tools, workflows; And limitations -- ## Tools https://wiki.debian.org/CopyrightReviewTools I'm unfamiliar with most of them. I'm only describing the two I'm familiar with. Both licensecheck (Jonas) and debmake (Osamu) do template/regex matching. ## workflows uploader: ??? there doesn't seem to be a standard process to generate debian/copyright for all uploaders. I personally do `licensecheck -r --deb-machine . > debian/copyright` and manually tweak the content. ftp-master: possibly manually reviewing with MC + custom plugin I didn't follow the recommended way. I use `ranger` (vim keybinding, fluent file browsing with preview panel) for reviewing packages on ftp-master.d.o. ## Limitations * Tree structure is always missing (and actually not possible to present) in debian/copyright. When reviewing other's NEW package as trainee, I feel torturous to locate the license information for a single file in debian/copyright. * Tree structure is always missing. after importing a new upstream release with significant directory layout change, it will be inconvenient to locate the parts of debian/copyright should be updated. Things will become more complex when new licenses/copyrights emerged. * licensecheck dumps garbage when it encounters a binary file, e.g. PNG image. This is not a BUG, as ftp-masters indeed checks the possible metadata in a binary file to make sure whether there is extra copyright/license info. But this is something needs to be improved... * Generic file browsers are not designed for our special purpose, neither does the commercial tools. * etc. My idea --- ## Motivations License reviewing is certainly inevitable. Even if we can improve the efficiency of this process a tiny bit, it will greatly improve the efficiency of the community on the specific task we are talking about. I have a couple of other motivations but the above one is already strong enough. ## Core The core of my idea is a tree-structured intermediate representation (IR) for the "license reviewing tree". The IR is basically a directory tree with annotations on the file nodes. The IR can be stored as a, say, JSON file. To build such an tree-shaped IR, we need a couple of "backend" tools for checking the copyright & license info for a SINGLE file. Such "backend" includes but not limited to: * `licensecheck`. Given a file FILE, `licensecheck FILE` produces the license name. * `grep` or `ripgrep`. For example, `rg -i copyright FILE` always works well. * "neighbor". For example, given a source file "F/I/L/E" without any copyright & license info, looking for F/I/L/LICENSE, F/I/LICENSE, ..., etc like git does for the ".git" directory will help. The formated+filtered output of any combination of these backends can be attached to the corresponding IR. In contrast, a "frontend" tool is also needed for dealing with such IR in a higher level. My imagined "frontend" tool is a `ranger`-like file browser with specific designs. * the user can choose what backend(s) to use. If none is chosen, the frontend tool falls back into a general file browser with a preview panel. * the frontend invokes various backend to generate a template IR, and store it to debian/copyright.json. No wildcard or regex in file path is allowed in this file. * when viewing files, the suggestions from various backends are shown. the user could choose to accept of override the suggestion. These choices will also be recorded in the json file. Of course, when various backends do not agree with each other, the user has to override the suggestion, and manually annotate the node. * when finished reviewing/annotating the whole directory tree,
Bug#947399: ITP: yuzu -- Nintendo Switch Emulator
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: yuzu Version : git HEAD * URL : https://yuzu-emu.org/ * License : GPL Programming Lang: C++ Description : Nintendo Switch Emulator will be maintained under the games team.
Bug#947160: ITP: sk1 -- advanced vector graphics editor
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: sk1 Version : 2.0~rc4 * URL : https://sk1project.net/sk1/ * License : GPL-3 Programming Lang: Python Description : advanced vector graphics editor Inkscape is good and I like it. This alternative may look good as well. I'll decide whether to really package it after trying it out.
difficulty in understanding options in init-system-GR
Hi folks, I went through the options in the init-system-GR[1] but I feel difficult to tell the subtle differences between the options. I'd like to ask for some hints here. Option F is distince from the others as it clearly emphasizes "deeper systemd integration". Option E puts equal weights to both systemd and non-systemd solutions. So let's skip the two. The rest options, i.e. B A D H G, look nearly the same to me: "first tier support for systemd, second tier for others" The only difference I noted is the bug severity for non-systemd support. In which way(s) are options (B, A, D, H, G) different from each other? A couple of keywords should be helpful enough to differentiate them. Thanks in advance. P.S. It could be extremely useful if a straightforward comparison table like [this](https://en.wikipedia.org/wiki/Comparison_of_file_systems) is available. But it isn't. I'd kindly suggest the future GR initiators make some comparison tables as long as options with subtle differences appear. [1] https://www.debian.org/vote/2019/vote_002
Bug#945918: ITP: libflame -- High-performance object-based library for DLA computations
Package: wnpp
Severity: wishlist
Owner: Mo Zhou
X-Debbugs-CC: debian-scie...@lists.debian.org
* Package name: libflame
Version : x.y.z
Upstream Author : Field G. Van Zee / UT Austin
* URL : https://github.com/flame/libflame
* License : BSD-3-Clause
Programming Lang: C
Description : High-performance object-based library for DLA computations
LAPACK implementation. Faster than existing ones (free) in our archive.
{BLAS=openblas/blis LAPACK=libflame} shows a significant
speed gain in sgesvd (single-precision SVD) compared to
{BLAS=openblas/blis LAPACK=netlib/openblas}
I didn't compare it with MKL. It's unnecessary.
Bug#945834: ITP: trisycl -- Generic system-wide modern C++ for heterogeneous platforms with SYCL from Khronos Group
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: trisycl Version : x.y.z Upstream Author : Name * URL : https://github.com/triSYCL/triSYCL * License : LLVM/UIUC Programming Lang: C++/SYCL Description : Generic system-wide modern C++ for heterogeneous platforms with SYCL from Khronos Group Seemed to me that intel is going to use SYCL for both their FPGA and the GPUs[1]. So I think it could be interesting the investigate the sycl implementations. An informative diagram about SYCL implementations: https://github.com/illuhad/hipSYCL#why-use-hipsycl-over-raw-cudahip This ITP means that I'm interested in investigating the SYCL implementations. But it doesn't mean that I'll really begin the debianization soon. [1] https://github.com/intel/BaseKit-code-samples.git
GPU-ready (with free driver) buildd/CI/porterbox?
Hi folks, Just a bold thought. Two Debian developers intended to package basic libraries of the AMD ROCm (freesoftware-licensed) [1], but both of them don't have access to such hardware. Given that there are also a number of packages in debian with OpenCL enabled, I'm wondering wether a shared buildd/CI/porterbox with GPU[2] could be helpful? Disclaimer: I have zero experience about AMD GPU, and I don't know how good the free driver is. [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=943556#20 [2] I mean AMD GPU running with free driver. None of nvidia's business here.
Re: Python 2 removal in sid/bullseye: Progress and next steps
On Tue, Nov 12, 2019 at 10:07:26AM +0900, Norbert Preining wrote: > Yes, we have a lot of python scripts, and some have been written more > than 10 years ago, and there is no maintainer anymore. > ... > And *we* at the TeX side prefer not to simply throw out old stuff, see > above. How many users still use these unmaintained scripts? Could you please provide us a reference/estimated popcon so we can better understand the situation of texlive in terms of py2rm.
Bug#944471: ITP: ccv -- C-based Computer Vision Library
Package: wnpp Severity: wishlist Owner: Mo Zhou * Package name: ccv Version : x.y.z Upstream Author : Name * URL : https://github.com/liuliu/ccv * License : BSD-3 Programming Lang: C Description : C-based Computer Vision Library OpenCV dropped support for their C interface. Maybe some users will use this one in their C project instead.
Re: Usage of DEP5
On Thu, Nov 07, 2019 at 08:33:23AM +0100, Ole Streicher wrote: > Andreas Tille writes: > > I would love to see another discussion here to reach more uniformity in > > Debian packaging and rise importance of DEP5 by recommending it in > > Debian Policy. > > I would really support that. A recommendation does not mean that there > may be some exceptional cases where DEP5 is not applicable, although I > think that DEP5 is flexible enough to cope with almost all cases. +1. Only with machine readable format do we have a chance for automation.
Re: MBF: don't build against libatlas3-base if possible
Hi Ian, On Wed, Oct 30, 2019 at 01:17:20PM +, Ian Jackson wrote: > Hi. I would like to congratulate you on your work so far. I am no > expert on this area but from what I read, I want to encourage you :-). :-) > I think it's important to give the MBF recipients a clear set of > instructions on what to change. And to assume that they may not know > what they are doing :-). As discussed here, we need to analyze the rdeps of libatlas3-base case by case because there may be some special situations: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=943712 (/me proposed to drop libcblas.so from libatlas3-base) Following that bug, I've studied two packages and provided patches for them: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=943828 (altree) https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=943827 (meep) So generally the maintainer should change the B-D from < libatlas-base-dev > to < libblas-dev >. Sometimes the maintainer also needs to patch the build system to force it link against -lblas instead of -lcblsa (see 943828) for example. > Your email gives clear instructions on what change to the .deb is > wanted: > > [...] Ideally any program that needs BLAS/CBLAS ABIs should link > > against libblas.so.3 instead of libcblas.so.3 (note the char "c" in > > middle) > > But it is not clear to me how this should be achieve in a source > package which currently build-depends on libatlas3-dev. Can the > build-dependency simpy be swapped out ? If so, to what ? Or are > other changes typically needed ? * A build system that supports many different BLAS implementations should be able to deal with -- the maintainer just needs to change the B-D. * When the build system unconditionally links against -lcblas, the maintainer needs to patch it into -lblas * If the package is sensitive to BLAS precision, or BLAS speed, the maintainer may optionaly add this for the binary package: Recommends: libopenblas-base | libblis2 | libatlas3-base | libmkl-rt | libblas3 | libblas.so.3 If the reference blas (i.e. libblas3) triggers FTBFS due to numerical precision problem, building with a high performance BLAS is fine, but the maintainer should carefully check the resulting package and avoid unexpected linkage such as -lcblas, -latlas, -lblis, -lopenblas, -lmkl, etc. * If the program needs implementation-specific features, no change would be required. For example, currently Julia needs some specific features of openblas (openblas_set_num_threads etc) although it can be disentangled with some effort.. > (Please forgive me for writing this mail without having actually > looked at any affected source package.) Nevermind. The MBF title looks short but the underlying problem is not that simple and straightforward. > > @Andreas, can we add this point to science-team policy? > > I will leave this to Andreas. But if Andreas and the science-team > agree with you, then this is clearly a candidate for a lintian > warning. You probably want to file a bug against lintian. I didn't even think of adding a lintian check for this purpose. We can add two lintian WARNINGs: W: linkage-against-specific-blas-lapack-implementation triggered when detected any specific BLAS/LAPACK implementation in shlibs:Depends . The package should link against the standard blas/lapack: libblas3 | libblas.so.3, liblapack3 | liblapack.so.3, In order to enable the update-alternative mechanism. W: linkage-against-libcblas triggered when libcblas.so.* has been found in shlibs:Depends. that indicates the maintainer may have fallen into a trap. I'll raise a discussion about this on -science and take further action according to the result. > Please keep up the good work. > > Regards, > Ian. > > -- > Ian JacksonThese opinions are my own. > > If I emailed you from an address @fyvzl.net or @evade.org.uk, that is > a private address which bypasses my fierce spamfilter.
Re: Summary: BLAS/LAPACK Ecosys Massive Update
On 2019-10-29 09:20, Benda Xu wrote: > Does that imply the blas64 inside src:julia could be finally unbundled? Further investigation needed. I'm still thinking of a better solution than compiling a decicated openblas from src:openblas for julia.
Summary: BLAS/LAPACK Ecosys Massive Update
Hi fellow developers, A good news especially for Debian scientific computing users. I shall call it a massive update, even if the whole update was decomposed into many tiny steps where some of them had already been finished 1 year ago. BLAS/LAPACK are two typical and classical dense linear algebra libraries, directly or indirectly used by most scientific computing software that involve vector & matrix operations, or basic linear algebra problems such as solving linear systems, solving linear least-square problems, or matrix factorization. A glance at the popcon and their reverse depends would demonsrate their importance: popcon > 0.1 million, and has notable rdeps such as numpy, scipy, octave, arpack, (julia), etc. TIPS: the typical performance bottle neck in a deep neural network is matrix multiplication (already mentioned here[1]). And that can be done by BLAS. For long time Debian's BLAS/LAPACK ecosystem lack a feature -- they should be compiled with different configurations due to different user demands. For example, a pthread application doesn't want to be linked against the library compiled in openmp settings; a supercomputer user may want to deal with a super large numerical array that cannot be indexed by 32-bit integer. That's the two massive updates we are going to talk about: (1) different threading flavours for BLAS/LAPACK implementations (2) different indexing (32bit v.s. 64bit) ... In 2018 I introduced intel-mkl (the fastest CPU-based BLAS/LAPACK implementation on x86 architecture, yet non-free) to our archive and registered it as an alternative of BLAS/LAPACK, and as the first alternative of BLAS64/LAPACK64. FYI, intel-mkl's magical runtime dispatching library libmkl_rt.so supports both (32-bit, 64-bit) indexing, and supports (gnu openmp, intel/llvm openmp, tbb, serial/sequential) threadding. Then I introduced blis (second fastest CPU-based free impl). It's a new package so it could be quite convenient for me to experiment the imagined ecosystem update: FYI: blis: (32bit,64bit) x (openmp,pthread,serial) Note that blis only provides BLAS/CBLAS implementation, not including LAPACK. Next I took part in GSoC2019/Gentoo, and introduced a BLAS/LAPACK runtime switching mechanism for Gentoo, fixing a long standing BLAS/LAPACK obstacle for Gentoo community: https://wiki.gentoo.org/wiki/Blas-lapack-switch After finishing the GSoC, I started to patch src:lapack which is the pivot package for the whole BLAS/LAPACK ecosys: FYI: src:lapack, (32bit,64bit)x(serial) It is the standard Fortran implementation. No multi-threading support. And subsequently src:openblas (fastest, free impl) FYI: openblas (32bit,64bit)x(pthread,openmp,serial) Just cleared NEW queue (experimental) several hours ago. So far all of my planned updates are basically finished (as long as openblas is uploaded to sid). We still need some time to test the features and make everything stablize, but our BLAS/LAPACK ecosys has already entered a new era. Notes for Debian Users == I recommend the following BLAS/LAPACK combinations, sorted by computation speed: * BLAS=libmkl-rt, LAPACK=libmkl-rt (non-free) * BLAS=openblas, LAPACK=openblas * BLAS=blis, LAPACK=lapack * BLAS=atlas, LAPACK=? (not tested) * BLAS=blas, LAPACK=lapack (standard, slow) I wrote a tool named "rover", a TUI frontend of update-alternatives, which is already present in stable/unstable. Initially it was written in order to conveniently debugging alternatives, however it is also useful enough and users can switch alternatives with it conveniently. Notes for Debian Developers === If you maintain packages depending on libblas.so|libblas-dev, or liblapack.so|liblapack-dev: Q: should I link my package against the 64-bit variant instead? A: don't do that if you are not sure. However, please investigate this problem if your package is intended for supercomputing or non-trivial numerical computation. ... I'll write a documentation about this whole thing and put it somewhere, to avoid a lengthy mail ... please look forward to the future update.. Currently we have this brief wiki page: https://wiki.debian.org/DebianScience/LinearAlgebraLibraries Acknowledgement === Many thanks to Sébastien Villemot who had been maintaining Debian's BLAS/LAPACK ecosys for many years. He reviewed nearly every of my step mentioned in this main, and helped me find many bugs and problems. FYI: Once upon a time when I was not a debian dev but merely a user, I had been deeply impressed by Sébastien's openblas packaging because it contains some carefully designed rules targets for the user to recompile the package locally, and that helped me at that time. That experience definitely further motivated my wish to become a DD. Thank you. Many thanks to Debian Science team that also provided a number of helpful feedbacks and suggestions. Many thanks to Aron
