Hi Steffen, On Mon, Jun 08, 2020 at 03:17:39AM +0200, Steffen Möller wrote: > Hello, > > pore-c has a series of dependencies that are of interest for Debian as a > whole and still missing > > python3-pyarrow > python3-streamz, > python3-pyranges, > python3-pairtools, > python3-ncls, > python3-intake
OK, so we can spent the time to package this until we have sorted out the LFS issue. Please urgently remember: If the Python tool is somehow biology or since related it makes perfectly sense to keep it inside Debian Med team. We are way better to do QA once it is in our scope and error reports go to some mailing list that is watched by our team closely. The pure fact that a module is written in Python is no good reason to to the packaging in Python team and as far as I understood members of the Python team they want only those packages there that are *closely watched* by members of the DPMT - which is not the case for our Debian Med packages. So please stay here if it makes any sense and the package is not extremely generic. > The package builds but does not run without these. In an attempt to push > what I have, I ran into > > remote: GitLab: LFS objects are missing. Ensure LFS is properly set up > or try a manual "git lfs push --all". > To salsa.debian.org:med-team/pore-c.git > ! [remote rejected]master -> master (pre-receive hook declined) > ! [remote rejected]pristine-tar -> pristine-tar (pre-receive hook declined) > ! [remote rejected]upstream -> upstream (pre-receive hook declined) > error: failed to push some refs to 'salsa.debian.org:med-team/pore-c.git' > $ git lfs push --all > Specify a remote and a remote branch name (`git lfs push origin master`) > $ git lfs push origin master > Locking support detected on remote "origin". Consider enabling it with: > $ git config > lfs.https://salsa.debian.org/med-team/pore-c.git/info/lfs.locksverify true > LFS upload failed:cts: 0% (0/23), 0 B | 0 B/s > (missing) tests/data/NlaIII_run01_GRCh38.haplotagged.bam > (1214675dd07b5a072ecbe9bf22ae7513c6f2c6b1435a86c0b8b93e6e85da6293) > (missing) tests/data/NlaIII_GRCh38.contacts.parquet/part.0.parquet > (94b3ebd6d94f10a2de3093d46473a51c003cd41713c9b3712d56ab30b676e193) > (missing) tests/data/NlaIII_GRCh38.concatemers.parquet/_metadata > (91d16d1ec1483db8b8080179ba9b485dce5bb625110768c23c348df6bef19e60) > (missing) tests/data/GM12878_NlaIII_reads.tsv > (7ce4529be82183332f9f3e37fa85f19a0a2aa5afa887f742145016bb2edcca53) > (missing) tests/data/GRCh38.chromsizes > (50e35247e1305e2a0a1fd1b7c07ff3576cec9389382d611c19eb0055633fc82b) > (missing) tests/data/GM12878.phased.conf.vcf.gz > (a7efa2fe2641ac2c76c0596ad628f90d9a8531856115f8b840379e95ee0709c8) > (missing) tests/data/GM12878_NlaIII_reads.fq.gz > (9d722847361b9c2d037d50d7e9cd126c958f5a0f689694fe5dfef37ae3ac1c4a) > (missing) tests/data/GRCh38.fasta.gz.gzi > (be5698ffc76f644f9483d04b3d58c8c2c7f0ba6a1fd37e78ac5eb99f5901e4f3) > (missing) tests/data/NlaIII_GRCh38.contacts.parquet/_common_metadata > (950401c687f40c995d415799583fa38ed1e063ceddd500d77c182388dda35d00) > (missing) > tests/data/NlaIII_run01_GRCh38.align_table.parquet/_common_metadata > (82b94a7abce2fe49c8321de6ce5cb127c832e6de3dd74aa1c0e756098ab26e86) > (missing) tests/data/NlaIII_GRCh38.vd.fragments.parquet/_metadata > (b97516f9b93688053d27efe7f604fde9573e63444354e42d31507922df8ca3b1) > (missing) tests/data/GRCh38.fasta.gz.fai > (cbd68404c124a5e3ee4e4335bf907cbd0db00b1a077d695042495260716dfd9b) > (missing) tests/data/GRCh38.fasta.gz > (79582e1bbeec718000027048849a6413ee576f17c1d9aa877211db1f8d6a7b1b) > (missing) tests/data/NlaIII_run01_GRCh38.read_sort.bam > (b451a851e775d7c364eb844feafa1a0054939be6035b0e6cfb808d2e13db2f82) > (missing) tests/data/test_ns.sam > (d9a0a31551e5cec6444e9fbfb6b87f0b644ef5a4ab9bad09329a856eb39160de) > (missing) tests/data/GM12878.phased.conf.vcf.gz.tbi > (0db1dcb337cfe2d23dd6d2369b49ec1414961ee2305e635ca6af7289aacf16ab) > (missing) tests/data/NlaIII_GRCh38.contacts.parquet/_metadata > (83aaf5e4f09b9cafa30446bd3f8016164c450d23ba840cbd62bcef55049e3356) > Uploading LFS objects: 0% (0/23), 0 B | 0 B/s, done. > (missing) tests/data/NlaIII_GRCh38.concatemers.parquet/part.0.parquet > (24ce696c45a21106d36ca97278f49ac707c1d77b97f86c77408ae0ee42c9905f) > (missing) tests/data/NlaIII_run01_GRCh38.align_table.parquet/_metadata > (42660830367bfa83ef9b4a5a501bdf0e19fd6c3a826210e8098e297443aad04e) > (missing) tests/data/NlaIII_GRCh38.vd.fragments.parquet/part.0.parquet > (6684beefe4fa555e3ef5ba2dca13769cda043424c08ac8f2b647471d7b587c80) > (missing) > tests/data/NlaIII_run01_GRCh38.align_table.parquet/part.0.parquet > (8fecdb8434662f0c0892dcdd103078ff2c48c96f40cc3546a9e76638b9c42740) > (missing) > tests/data/NlaIII_GRCh38.concatemers.parquet/_common_metadata > (ec1a8f121551b7d0bccc8f2eb4097742029b30068313dc15a932b701e4d91b0c) > (missing) > tests/data/NlaIII_GRCh38.vd.fragments.parquet/_common_metadata > (2902630654ed49ea1ea6648dc21bba0c20b499fa6182a6b2ec6b531b7564cb6e) > hint: Your push was rejected due to missing or corrupt local objects. > hint: You can disable this check with: 'git config > lfs.allowincompletepush true' > > Well, kind of curious I did enable it, but could not push --all. After > also pushing upstream, I went for pristine-tar without lsf > > > $ git push --set-upstream origin pristine-tar > Enumerating objects: 4, done. > Counting objects: 100% (4/4), done. > Delta compression using up to 2 threads > Compressing objects: 100% (3/3), done. > Writing objects: 100% (4/4), 3.11 KiB | 3.11 MiB/s, done. > Total 4 (delta 0), reused 0 (delta 0), pack-reused 0 > To salsa.debian.org:med-team/pore-c.git > * [new branch] pristine-tar -> pristine-tar > Branch 'pristine-tar' set up to track remote branch 'pristine-tar' from > 'origin'. > moeller@steffen-laptop-debian:~/git/med-team/pore-c/pore-c$ git checkout > master > Downloading tests/data/GM12878.phased.conf.vcf.gz (516 KB) > Error downloading object: tests/data/GM12878.phased.conf.vcf.gz > (a7efa2f): Smudge error: Error downloading > tests/data/GM12878.phased.conf.vcf.gz (a7efa2fe2641ac2c76c0596ad628f90d9a853 > 1856115f8b840379e95ee0709c8): > [a7efa2fe2641ac2c76c0596ad628f90d9a8531856115f8b840379e95ee0709c8] > Object does not exist on the server or you don't have permissions to > access it: [404] > Object does not exist on the server or you don't have permissions to > access it > > Errors logged to > /home/moeller/git/med-team/pore-c/pore-c/.git/lfs/logs/20200608T030628.335563029.log > Use `git lfs logs last` to view the log. > error: external filter 'git-lfs filter-process' failed > fatal: tests/data/GM12878.phased.conf.vcf.gz: smudge filter lfs failed > moeller@steffen-laptop-debian:~/git/med-team/pore-c/pore-c$ git branch > master > * pristine-tar > upstream > > Hm. Ok. This is all new to me. I admit I had to deal with lfs the first time in my first attempt with gatk which was stalled. There the download tarball contained a really huge file that would require git lfs. However, I had to do again with git lfs in flappie and here the case was different - and may be is the same as in your case. The flappie download tarball did *not* contain large files but just broken git lfs references. Pretty short files referencing some Git location. That's pretty useless in a tarball - no idea how to prevent this at tarball creation time. I simply excluded these files in https://salsa.debian.org/med-team/flappie/-/blob/master/debian/copyright Since all are test data we can find some other means if it is about testing later. If I'm not missing anything in your log its also only about test data. If you ask me you could try the same, remove the files - which in case of non-working references should be cheap anyway - and recreate the git repository. > I know about the large file support, but > why is this enforced on me when working with a regular tarball that is > not even big (most likely because of those special files pointing to > external data). I noticed that on stable `gbp import-orig` is including the tarball - just tested it since I suspected some bug in gbp from unstable. However, its not a bug but a feature: We do not want non-working references in our repository. Thus I removed this and our repository is at least consistent while lacking **references** to test data. The test data itself are missing in the tarball anyway. Your description smells pretty much like the same I have observed in flappie. > My immediate response would be to remove the test data > and find other ways to perform the testing. So we perfectly agree here. :-) Kind regards Andreas. -- http://fam-tille.de
