Re: [gentoo-user] emerge --sync fails with a python error
On 15.05.23 16:41, Matt Connell wrote: On Mon, 2023-05-15 at 16:24 +0200, Dan Johansson wrote: RuntimeError: OpenPGP signature not found on Manifest It sounds like your sync is hitting a mirror that is currently broken. Are you using a defined mirror list or letting it auto-select? As far as I can tell, portage is using "auto-select". in /etc/portage/make.conf I do not have GENTOO_MIRRORS set. -- Dan Johansson *** This message is printed on 100% recycled electrons! ***
Re: [gentoo-user] emerge --sync fails with a python error
On Mon, 2023-05-15 at 16:24 +0200, Dan Johansson wrote: > RuntimeError: OpenPGP signature not found on Manifest It sounds like your sync is hitting a mirror that is currently broken. Are you using a defined mirror list or letting it auto-select?
Re: [gentoo-user] emerge --sync keeps failing
On 8/2/21 2:01 PM, Michael wrote: On Monday, 2 August 2021 12:10:12 BST n952162 wrote: On 8/1/21 8:32 PM, Michael Orlitzky wrote: On Sun, 2021-08-01 at 17:32 +0200, n952162 wrote: * Verifying /var/db/repos/gentoo/.tmp-unverified-download-quarantine ...!!! Manifest verification failed: Manifest mismatch for metadata/news/Manifest I've raised this question before and the only useful answer I got was to keep trying On the off chance that something is screwy on the remote end, you can always use "emerge-webrsync" and delay the problem until next time. Otherwise, I would say check "dmesg" for disk errors, but if it's happening on two machines that's a lot less likely. I have this problem every month. Why does it fail? Is it just a timeout because my network is slow? Can that be tweaked? I get this problem over here, but on rare occasions. Leaving it for half a day usually fixes it. Have you tried a different rsync mirror? You can use 'mirrorselect -i -r' for this task. Unfortunately, mirrorselect doesn't seem to work anymore: configparser.MissingSectionHeaderError: File contains no section headers. It seems to have lost track of whether it's editing make.conf or repos.conf. I ran it with the -o option, to a tempfile and tried to put that into /etc/portage/repos.conf, I think, but that didn't work either. Next, I manually editted /usr/share/portage/config/repos.conf and put that line in where the old sync-uri was defined. But that didn't work either. Now, I've created a new directory /etc/portage/repos.conf/ and moved the file generate by the -o of mirrorselect into that as gentoo.conf and added a "[gentoo]" section head. It MIGHT be working now.
Re: [gentoo-user] emerge --sync keeps failing
On Mon, 2021-08-02 at 13:10 +0200, n952162 wrote: > > > > I have this problem every month. Why does it fail? Is it just a > timeout because my network is slow? Can that be tweaked? > I'm not really sure. I've seen it fail in the past due to bad memory or a dying hard drive, but it also "just happens." It might be the upstream mirror? I'd have to investigate precisely what is failing.. and that would take longer than re-syncing a few times every couple of months.
Re: [gentoo-user] emerge --sync keeps failing
On Tuesday, 3 August 2021 11:49:23 BST Neil Bothwick wrote: > On Tue, 03 Aug 2021 09:55:46 +0100, Michael wrote: > > Anyhow, I recall using git to sync a live ebuild - from some repo > > and portage started downloading gigabytes of cruft. Presumably whole > > decades of old commits I didn't need or have space for. I subsequently > > discovered I had to set "EGIT_CLONE_TYPE=shallow" in make.conf, but I > > can't find this variable in the man page now. I suppose/hope portage > > using git will only download more recent commits? > > It does, you get just the current state of the tree by default, it's just > orders of magnitude faster, even compared with using rsync with a local > mirror. And I don't have to worry abut syncing too often and upsetting > infra as I'm syncing from github. I see, thanks Neil. Does it now check sigs of downloaded data, like rsync does? signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] emerge --sync keeps failing
On Tue, 03 Aug 2021 09:55:46 +0100, Michael wrote: > Anyhow, I recall using git to sync a live ebuild - from some repo > and portage started downloading gigabytes of cruft. Presumably whole > decades of old commits I didn't need or have space for. I subsequently > discovered I had to set "EGIT_CLONE_TYPE=shallow" in make.conf, but I > can't find this variable in the man page now. I suppose/hope portage > using git will only download more recent commits? It does, you get just the current state of the tree by default, it's just orders of magnitude faster, even compared with using rsync with a local mirror. And I don't have to worry abut syncing too often and upsetting infra as I'm syncing from github. -- Neil Bothwick Talk is cheap because supply exceeds demand. pgpeCI718v3Nf.pgp Description: OpenPGP digital signature
Re: [gentoo-user] emerge --sync keeps failing
On Tuesday, 3 August 2021 02:28:09 BST John Covici wrote: > On Mon, 02 Aug 2021 18:42:29 -0400, > > Michael wrote: > > > > On Monday, 2 August 2021 22:17:35 BST Neil Bothwick wrote: > > > On Mon, 02 Aug 2021 13:01:40 +0100, Michael wrote: > > > > > I have this problem every month. Why does it fail? Is it just a > > > > > timeout because my network is slow? Can that be tweaked? > > > > > > > > I get this problem over here, but on rare occasions. Leaving it for > > > > half a day usually fixes it. Have you tried a different rsync mirror? > > > > You can use 'mirrorselect -i -r' for this task. > > > > > > I used to see this from time to time, but then I switched to using git > > > instead of rsync and haven't seen it again. There's the added bonus that > > > syncing is MUCH faster too. > > > > Last time I looked at git for portage it was not not using gpg to verify > > the downloaded data - has this feature been added since then? > > I don't know when you last looked, but I see it checking the signature > when downloading from git. It was some inordinate time ago - can't recall when exactly. It was after portage started using gpg with rsync and quarantined any fetched files after they were downloaded until their signatures were validated. As I understood at the time the difference with using git was it would only check the top commit in the repository had a valid signature, but not what eventually landed at the local PC. Also, even to do this check it required you have enabled "sync-git-verify-commit-signature = yes" - but according to 'man portage' it defaults to "no". Anyhow, I recall using git to sync a live ebuild - from some repo and portage started downloading gigabytes of cruft. Presumably whole decades of old commits I didn't need or have space for. I subsequently discovered I had to set "EGIT_CLONE_TYPE=shallow" in make.conf, but I can't find this variable in the man page now. I suppose/hope portage using git will only download more recent commits? signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] emerge --sync keeps failing
On Mon, 02 Aug 2021 18:42:29 -0400, Michael wrote: > > [1 ] > On Monday, 2 August 2021 22:17:35 BST Neil Bothwick wrote: > > On Mon, 02 Aug 2021 13:01:40 +0100, Michael wrote: > > > > I have this problem every month. Why does it fail? Is it just a > > > > timeout because my network is slow? Can that be tweaked? > > > > > > I get this problem over here, but on rare occasions. Leaving it for > > > half a day usually fixes it. Have you tried a different rsync mirror? > > > You can use 'mirrorselect -i -r' for this task. > > > > I used to see this from time to time, but then I switched to using git > > instead of rsync and haven't seen it again. There's the added bonus that > > syncing is MUCH faster too. > > Last time I looked at git for portage it was not not using gpg to verify the > downloaded data - has this feature been added since then? I don't know when you last looked, but I see it checking the signature when downloading from git. -- Your life is like a penny. You're going to lose it. The question is: How do you spend it? John Covici wb2una cov...@ccs.covici.com
Re: [gentoo-user] emerge --sync keeps failing
On Monday, 2 August 2021 22:17:35 BST Neil Bothwick wrote: > On Mon, 02 Aug 2021 13:01:40 +0100, Michael wrote: > > > I have this problem every month. Why does it fail? Is it just a > > > timeout because my network is slow? Can that be tweaked? > > > > I get this problem over here, but on rare occasions. Leaving it for > > half a day usually fixes it. Have you tried a different rsync mirror? > > You can use 'mirrorselect -i -r' for this task. > > I used to see this from time to time, but then I switched to using git > instead of rsync and haven't seen it again. There's the added bonus that > syncing is MUCH faster too. Last time I looked at git for portage it was not not using gpg to verify the downloaded data - has this feature been added since then? signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] emerge --sync keeps failing
On Mon, 02 Aug 2021 13:01:40 +0100, Michael wrote: > > I have this problem every month. Why does it fail? Is it just a > > timeout because my network is slow? Can that be tweaked? > > I get this problem over here, but on rare occasions. Leaving it for > half a day usually fixes it. Have you tried a different rsync mirror? > You can use 'mirrorselect -i -r' for this task. I used to see this from time to time, but then I switched to using git instead of rsync and haven't seen it again. There's the added bonus that syncing is MUCH faster too. -- Neil Bothwick WinErr 019: User error - Not our fault. Is Not! Is Not! pgpp86ID1spVr.pgp Description: OpenPGP digital signature
Re: [gentoo-user] emerge --sync keeps failing
On 8/2/21 2:01 PM, Michael wrote: On Monday, 2 August 2021 12:10:12 BST n952162 wrote: On 8/1/21 8:32 PM, Michael Orlitzky wrote: On Sun, 2021-08-01 at 17:32 +0200, n952162 wrote: * Verifying /var/db/repos/gentoo/.tmp-unverified-download-quarantine ...!!! Manifest verification failed: Manifest mismatch for metadata/news/Manifest I've raised this question before and the only useful answer I got was to keep trying On the off chance that something is screwy on the remote end, you can always use "emerge-webrsync" and delay the problem until next time. Otherwise, I would say check "dmesg" for disk errors, but if it's happening on two machines that's a lot less likely. I have this problem every month. Why does it fail? Is it just a timeout because my network is slow? Can that be tweaked? I get this problem over here, but on rare occasions. Leaving it for half a day usually fixes it. Have you tried a different rsync mirror? You can use 'mirrorselect -i -r' for this task. Ah, good idea.
Re: [gentoo-user] emerge --sync keeps failing
Michael wrote: > On Monday, 2 August 2021 12:10:12 BST n952162 wrote: >> On 8/1/21 8:32 PM, Michael Orlitzky wrote: >>> On Sun, 2021-08-01 at 17:32 +0200, n952162 wrote: * Verifying /var/db/repos/gentoo/.tmp-unverified-download-quarantine ...!!! Manifest verification failed: Manifest mismatch for metadata/news/Manifest I've raised this question before and the only useful answer I got was to keep trying >>> On the off chance that something is screwy on the remote end, you can >>> always use "emerge-webrsync" and delay the problem until next time. >>> >>> Otherwise, I would say check "dmesg" for disk errors, but if it's >>> happening on two machines that's a lot less likely. >> I have this problem every month. Why does it fail? Is it just a >> timeout because my network is slow? Can that be tweaked? > I get this problem over here, but on rare occasions. Leaving it for half a > day usually fixes it. Have you tried a different rsync mirror? You can use > 'mirrorselect -i -r' for this task. This is a very good idea. I had to switch a few months ago. The one I was using was either really busy or had other issues that slowed it to a crawl. It reminded me of dial-up days. I might add, I found some that didn't appear to be set up to sync with anymore. It appeared they removed Gentoo files from the server. Could be a glitch so I didn't report it. I synced last night, about 12 hours ago, without any problems. Dale :-) :-)
Re: [gentoo-user] emerge --sync keeps failing
On Monday, 2 August 2021 12:10:12 BST n952162 wrote: > On 8/1/21 8:32 PM, Michael Orlitzky wrote: > > On Sun, 2021-08-01 at 17:32 +0200, n952162 wrote: > >>* Verifying /var/db/repos/gentoo/.tmp-unverified-download-quarantine > >> > >> ...!!! Manifest verification failed: > >>Manifest mismatch for metadata/news/Manifest > >> > >> I've raised this question before and the only useful answer I got was to > >> keep trying > > > > On the off chance that something is screwy on the remote end, you can > > always use "emerge-webrsync" and delay the problem until next time. > > > > Otherwise, I would say check "dmesg" for disk errors, but if it's > > happening on two machines that's a lot less likely. > > I have this problem every month. Why does it fail? Is it just a > timeout because my network is slow? Can that be tweaked? I get this problem over here, but on rare occasions. Leaving it for half a day usually fixes it. Have you tried a different rsync mirror? You can use 'mirrorselect -i -r' for this task. signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] emerge --sync keeps failing
On 2021-08-02 09:20:19, n952162 wrote: > On 8/1/21 8:32 PM, Michael Orlitzky wrote: > > On Sun, 2021-08-01 at 17:32 +0200, n952162 wrote: > >> * Verifying /var/db/repos/gentoo/.tmp-unverified-download-quarantine > >> ...!!! Manifest verification failed: > >> Manifest mismatch for metadata/news/Manifest > >> > >> I've raised this question before and the only useful answer I got was to > >> keep trying > >> > > On the off chance that something is screwy on the remote end, you can > > always use "emerge-webrsync" and delay the problem until next time. > > > Does that install binary images? > No, it just tells portage to use HTTP instead of rsync. One big file is downloaded and verified instead of many little ones.
Re: [gentoo-user] emerge --sync keeps failing
On 8/1/21 8:32 PM, Michael Orlitzky wrote: On Sun, 2021-08-01 at 17:32 +0200, n952162 wrote: * Verifying /var/db/repos/gentoo/.tmp-unverified-download-quarantine ...!!! Manifest verification failed: Manifest mismatch for metadata/news/Manifest I've raised this question before and the only useful answer I got was to keep trying On the off chance that something is screwy on the remote end, you can always use "emerge-webrsync" and delay the problem until next time. Otherwise, I would say check "dmesg" for disk errors, but if it's happening on two machines that's a lot less likely. I have this problem every month. Why does it fail? Is it just a timeout because my network is slow? Can that be tweaked?
Re: [gentoo-user] emerge --sync keeps failing
On 8/2/21 9:20 AM, n952162 wrote: On 8/1/21 8:32 PM, Michael Orlitzky wrote: On Sun, 2021-08-01 at 17:32 +0200, n952162 wrote: * Verifying /var/db/repos/gentoo/.tmp-unverified-download-quarantine ...!!! Manifest verification failed: Manifest mismatch for metadata/news/Manifest I've raised this question before and the only useful answer I got was to keep trying On the off chance that something is screwy on the remote end, you can always use "emerge-webrsync" and delay the problem until next time. Does that install binary images? Otherwise, I would say check "dmesg" for disk errors, but if it's happening on two machines that's a lot less likely. There's plenty of space available. No movement in /var/log/messages during the attempt. I've had two fails already this morning.
Re: [gentoo-user] emerge --sync keeps failing
On 8/1/21 8:32 PM, Michael Orlitzky wrote: On Sun, 2021-08-01 at 17:32 +0200, n952162 wrote: * Verifying /var/db/repos/gentoo/.tmp-unverified-download-quarantine ...!!! Manifest verification failed: Manifest mismatch for metadata/news/Manifest I've raised this question before and the only useful answer I got was to keep trying On the off chance that something is screwy on the remote end, you can always use "emerge-webrsync" and delay the problem until next time. Does that install binary images? Otherwise, I would say check "dmesg" for disk errors, but if it's happening on two machines that's a lot less likely. There's plenty of space available.
Re: [gentoo-user] emerge --sync keeps failing
On Sun, 2021-08-01 at 17:32 +0200, n952162 wrote: > * Verifying /var/db/repos/gentoo/.tmp-unverified-download-quarantine > ...!!! Manifest verification failed: > Manifest mismatch for metadata/news/Manifest > > I've raised this question before and the only useful answer I got was to > keep trying > On the off chance that something is screwy on the remote end, you can always use "emerge-webrsync" and delay the problem until next time. Otherwise, I would say check "dmesg" for disk errors, but if it's happening on two machines that's a lot less likely.
Re: [gentoo-user] "emerge --sync" and "gemato" screwed
On Sat, Nov 14, 2020 at 12:20:16AM -0500, Walter Dnes wrote > = > > And before anyone asks, "emerge -pv1 portage" shows rsync-verify > is disabled... > > = > Calculating dependencies... done! > [ebuild R] sys-apps/portage-3.0.8::gentoo USE="(ipc) native-extensions > xattr -apidoc -build -doc -gentoo-dev -rsync-verify (-selinux) -test" > PYTHON_TARGETS="python3_7 (-pypy3) -python3_6 -python3_8 (-python3_9)" 0 KiB > = > > OK, so I emerged gemato, and a whole bunch of its dependencies, but > that doesn't help... It looks like I went about things exactly the wrong way. gemato should not be emerged directly. USE="rsync-verify" emerge -1 portage pulls in gemato and app-crypt/openpgp-keys-gentoo-release-20200704, and other dependencies. This would've prevented the mess about gemato not finding the keys. There's still the problem of the news item https://www.gentoo.org/support/news-items/2018-01-30-portage-rsync-verification.html -- Walter Dnes I don't run "desktop environments"; I run useful applications
Re: [gentoo-user] "emerge --sync" and "gemato" screwed
Walter Dnes wrote:
> Here's tail-end of a recent "emerge --sync" plus an update attempt...
>
> =
> Total bytes sent: 334.55K
> Total bytes received: 51.72M
>
> sent 334.55K bytes received 51.72M bytes 343.63K bytes/sec
> total size is 178.09M speedup is 3.42
> !!! Unable to verify: gemato-14.5+ is required
>
> Action: sync for repo: gentoo, returned code = 127
>
>
> [d531][root][~] emerge -pv --changed-use --deep --update @world
> * Last emerge --sync was 32d 14h 23m 26s ago.
>
> These are the packages that would be merged, in order:
>
> Calculating dependencies... done!
>
> Total: 0 packages, Size of downloads: 0 KiB
> =
>
> And before anyone asks, "emerge -pv1 portage" shows rsync-verify
> is disabled...
>
> =
> Calculating dependencies... done!
> [ebuild R] sys-apps/portage-3.0.8::gentoo USE="(ipc) native-extensions
> xattr -apidoc -build -doc -gentoo-dev -rsync-verify (-selinux) -test"
> PYTHON_TARGETS="python3_7 (-pypy3) -python3_6 -python3_8 (-python3_9)" 0 KiB
> =
>
> OK, so I emerged gemato, and a whole bunch of its dependencies, but
> that doesn't help...
>
> =
> [d531][root][~] emerge --sync
Syncing repository 'gentoo' into '/var/db/repos/gentoo'...
> * Using keys from /usr/share/openpgp-keys/gentoo-release.asc
> Traceback (most recent call last):
> File
> "/usr/lib/python3.7/site-packages/portage/util/_async/AsyncFunction.py", line
> 39, in _run
> result = self.target(*(self.args or []), **(self.kwargs or {}))
> File "/usr/lib/python3.7/site-packages/portage/sync/controller.py", line
> 165, in sync
> taskmaster.run_tasks(tasks, func, status, options=task_opts)
> File "/usr/lib/python3.7/site-packages/portage/sync/controller.py", line
> 65, in run_tasks
> result = getattr(inst, func)(**kwargs)
> File "/usr/lib/python3.7/site-packages/portage/sync/syncbase.py", line 338,
> in sync
> return self.update()
> File
> "/usr/lib/python3.7/site-packages/portage/sync/modules/rsync/rsync.py", line
> 145, in update
> with io.open(self.repo.sync_openpgp_key_path, 'rb') as f:
> FileNotFoundError: [Errno 2] No such file or directory:
> '/usr/share/openpgp-keys/gentoo-release.asc'
>
> Action: sync for repo: gentoo, returned code = 1
> =.
>
> Now what???
>
Well, I don't even see a gemato 14 in the tree here. This is what I show.
root@fireball / # equery list -p gemato
* Searching for gemato ...
[-P-] [ ] app-portage/gemato-15.2:0
[-P-] [ ~] app-portage/gemato-16.1:0
[IP-] [ ] app-portage/gemato-16.2:0
[-P-] [ -] app-portage/gemato-:0
root@fireball / #
It seems you are a bit behind on that package if I read that correctly.
I'm not sure how to work around it but I'll make this offer if it helps
and nothing else works. I have a binary for it that I can send you off
list. This is the info for the package and USE flags if they match your
needs close enough.
[ebuild R ] app-portage/gemato-16.2::gentoo USE="gpg -test -tools"
PYTHON_TARGETS="python3_7 (-pypy3) -python3_6 -python3_8 (-python3_9)"
I looked and the tarball is only ~102KBs. If you need something else,
I'd be happy to send you whatever you need. My DSL upload is slow but
it gets there, eventually. ;-)
Dale
:-) :-)
Re: [gentoo-user] Emerge --sync source
Mick wrote: > > I can think of 3 things, but more learned M/L contributors may add to these: > > 1. The SATA connection has come loose. With time and movement it can come > (slightly) adrift. Pushing it back in fully fixes this problem - also see No. > 2 below. > > 2. The physical connector's contacts are beginning to oxidise. Reseat the > SATA cable connectors both on the drive and any ribbons on the MoBo. This > usualy cleans any oxidisation. > I recently had to replace a SATA cable because it was causing errors on a drive. I tried reseating it because that usually works but in that case, it must have been a bad wire somewhere inside the cable. Maybe at some point it was bent around to much or something and was weak or almost broken. Once I replaced the cable, the drive started working correctly. I mention that to say this. Just try another cable even if only temporarily if you can. It's one sure way to know that isn't the problem at least. Dale :-) :-)
Re: [gentoo-user] Emerge --sync source
On Thursday, 7 March 2019 10:10:53 GMT Peter Humphrey wrote: > On Wednesday, 6 March 2019 16:31:27 GMT Laurence Perkins wrote: > > On Fri, 2019-03-01 at 10:12 +, Peter Humphrey wrote: > > > [OT] > > > Evidence is mounting that the Atom box is in terminal decline. I get > > > things like batches of files in the portage tree changing owner, and > > > then > > > when I correct that, long lists of supposedly locally changed ebuilds > > > preventing syncing. And when I boot weekly into its little rescue system > > > to backup the main system, the root filesystem remounts itself read-only > > > while tar is running. Smartd recognises the SSD and runs daily tests, > > > but > > > reports no errors. No amount of wiping and reinstalling has helped so > > > far. > > > > What filesystem are you running and how old is the SSD? That sounds > > like some of the symptoms EXT4 had on early generation flash media > > where its assumptions about what order writes would physically make it > > to the disk in were wrong, leading to corruption. > > The disk is a 64GB SanDisk SDSSDP device, which I bought five years ago to > replace a failed spinning disk. All partitions are ext4 except /boot, which > is ext2. > > > So unless it was working correctly at some point in the past, try a > > different filesystem. EXT3 or BTRFS didn't have the same problems. > > It was working just fine until recently. > > > If it's just that the SSD is failing, then get a new one before > > something important gets damaged and you have to redo the whole thing. > > Everything on it is disposable. > > The box is getting a bit long in the tooth: I bought it in November 2010. > It's a single-core, 32-bit Atom N270 (not N2700). It doesn't owe me > anything now, in spite of having cost £450 at the time. I don't know > whether it's worth throwing any more money at it. On the other hand, I see > Amazon are only asking for £20 for a small SSD. > > The repeatability of some of the errors it throws makes me question whether > the disk or something else is at fault. (What would cause a file system to > be remounted read-only in the middle of its work?) I can think of 3 things, but more learned M/L contributors may add to these: 1. The SATA connection has come loose. With time and movement it can come (slightly) adrift. Pushing it back in fully fixes this problem - also see No. 2 below. 2. The physical connector's contacts are beginning to oxidise. Reseat the SATA cable connectors both on the drive and any ribbons on the MoBo. This usualy cleans any oxidisation. 3. The AHCI driver is deploying energy saving measures (aka. Aggressive Link Power Management - ALPM). Check the output of: cat /sys/class/scsi_host/host*/link_power_management_policy If it doesn't say 'max_performance' you'll need to revisit your BIOS settings and also PCIEASPM settings in the kernel. 4. Finally, there is a chance the PSU is playing up. 1 & 2 above are more noticeable on spinning disks, but it is a matter of scale before SSDs are affected too. If BIOS, kernel settings and drivers were not altered recently, then 1 & 2 merit attention in the first instance. > I have a spare four-core, 64-bit Celeron box (I bought it for a purpose > that's gone away). I've been wondering what to do with it, so maybe it can > replace the Atom box. It's powerful enough to compile its own software, > whereas the Atom needs help. Whichever I use, its job will be as a server > of DNS, LAN mail, time and git. Maybe print too. Also it will fetch my > ISP's POP mail and serve it over IMAP to this box. > > > The self-test capability of storage media is almost universally > > horrible and you generally don't get a failure report until your data > > has already been lost. If your SMART output gives you the raw > > statistics on the device instead of just pass/fail then analyzing that > > usually gives a better indication of whether something is about to go > > wrong. > > It seems to report only pass/fail, so that's not much help. > > Decisions, decisions... Do short/long smartctl tests report no errors, assuming the disk comes with S.M.A.R.T. capability? -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Emerge --sync source
On Wednesday, 6 March 2019 16:31:27 GMT Laurence Perkins wrote: > On Fri, 2019-03-01 at 10:12 +, Peter Humphrey wrote: > > [OT] > > Evidence is mounting that the Atom box is in terminal decline. I get > > things like batches of files in the portage tree changing owner, and then > > when I correct that, long lists of supposedly locally changed ebuilds > > preventing syncing. And when I boot weekly into its little rescue system > > to backup the main system, the root filesystem remounts itself read-only > > while tar is running. Smartd recognises the SSD and runs daily tests, but > > reports no errors. No amount of wiping and reinstalling has helped so far. > > What filesystem are you running and how old is the SSD? That sounds > like some of the symptoms EXT4 had on early generation flash media > where its assumptions about what order writes would physically make it > to the disk in were wrong, leading to corruption. The disk is a 64GB SanDisk SDSSDP device, which I bought five years ago to replace a failed spinning disk. All partitions are ext4 except /boot, which is ext2. > So unless it was working correctly at some point in the past, try a > different filesystem. EXT3 or BTRFS didn't have the same problems. It was working just fine until recently. > If it's just that the SSD is failing, then get a new one before > something important gets damaged and you have to redo the whole thing. Everything on it is disposable. The box is getting a bit long in the tooth: I bought it in November 2010. It's a single-core, 32-bit Atom N270 (not N2700). It doesn't owe me anything now, in spite of having cost £450 at the time. I don't know whether it's worth throwing any more money at it. On the other hand, I see Amazon are only asking for £20 for a small SSD. The repeatability of some of the errors it throws makes me question whether the disk or something else is at fault. (What would cause a file system to be remounted read-only in the middle of its work?) I have a spare four-core, 64-bit Celeron box (I bought it for a purpose that's gone away). I've been wondering what to do with it, so maybe it can replace the Atom box. It's powerful enough to compile its own software, whereas the Atom needs help. Whichever I use, its job will be as a server of DNS, LAN mail, time and git. Maybe print too. Also it will fetch my ISP's POP mail and serve it over IMAP to this box. > The self-test capability of storage media is almost universally > horrible and you generally don't get a failure report until your data > has already been lost. If your SMART output gives you the raw > statistics on the device instead of just pass/fail then analyzing that > usually gives a better indication of whether something is about to go > wrong. It seems to report only pass/fail, so that's not much help. Decisions, decisions... -- Regards, Peter.
Re: [gentoo-user] Emerge --sync source
On 06/03/19 17:39, Alan Mackenzie wrote: > Up to now, I've never had a HDD or SDD fail on me. :-) I hope that > when this does eventually happen, I'll be prepared. I don't think I've had one of mine fail. I have, however, done recovery jobs on two drives that did fail that I managed to revive long enough to get the data off. And I currently have a second drive that is properly dead, whose owner has asked me to destroy it to make sure that nothing can be recovered off it (the first drive I had in that state was a 60 *G*B drive, which tells you that it was a long time ago). Drives are reliable. Drives do last a long time, and I think many drives have been upgraded before they failed. But nowadays, drives are so large that people don't fill them up and upgrade, so they are used a lot longer, and you're seeing them fail more often. I think I've handled five dead drives (friends and acquaintances) in my career and I'm sure others have seen a lot more. Cheers, Wol
Re: [gentoo-user] Emerge --sync source
Hello, Rich. I'd like to say hello again to everybody, just to mark that I'm still here and still using Gentoo, and thank people for (a lot of) help rendered in the past. My system, used mainly for SW development, has been stable and well behaved, with very occasional exceptions, for some while now. On Wed, Mar 06, 2019 at 11:51:47 -0500, Rich Freeman wrote: > On Wed, Mar 6, 2019 at 11:31 AM Laurence Perkins wrote: > > If it's just that the SSD is failing, then get a new one before > > something important gets damaged and you have to redo the whole thing. > IMO any kind of storage device should be treated as if it could fail > at any time without warning. You should have a plan for what you will > do WHEN this happens, not IF it happens. > If losing a storage device would result in you losing "something > important" then you're doing it wrong. > I keep all my spinning disks in some kind of RAID unless their > contents are completely expendable (ie I won't be upset if I > completely lose it). For SSDs I generally do frequent rsync or > zfs-send backups to a spinning disk - these are generally used for OS > data which doesn't change as much anyway, and the backups are quick > since they aren't large. If I had large SSDs I'd run them in some > sort of RAID. > And of course anything I consider really important gets backed up to > the cloud, encrypted. RAID is more about avoiding downtime and the > inconvenience of an offline restore. On my new box, from 2017-04, I don't have any spinning disks (other than the DVD burner). I just have a pair of NVMe SSDs in a RAID 1 configuration, with everything bar /boot mirrored. I back up once a week to one of two DVD+RWs (alternately), and encrypted to a "cloud" service (what used to be known as a "computer bureau"). Up to now, I've never had a HDD or SDD fail on me. :-) I hope that when this does eventually happen, I'll be prepared. > -- > Rich -- Alan Mackenzie (Nuremberg, Germany).
Re: [gentoo-user] Emerge --sync source
On Wed, Mar 6, 2019 at 11:31 AM Laurence Perkins wrote: > > If it's just that the SSD is failing, then get a new one before > something important gets damaged and you have to redo the whole thing. IMO any kind of storage device should be treated as if it could fail at any time without warning. You should have a plan for what you will do WHEN this happens, not IF it happens. If losing a storage device would result in you losing "something important" then you're doing it wrong. I keep all my spinning disks in some kind of RAID unless their contents are completely expendable (ie I won't be upset if I completely lose it). For SSDs I generally do frequent rsync or zfs-send backups to a spinning disk - these are generally used for OS data which doesn't change as much anyway, and the backups are quick since they aren't large. If I had large SSDs I'd run them in some sort of RAID. And of course anything I consider really important gets backed up to the cloud, encrypted. RAID is more about avoiding downtime and the inconvenience of an offline restore. -- Rich
Re: [gentoo-user] Emerge --sync source
On Fri, 2019-03-01 at 10:12 +, Peter Humphrey wrote: > On Thursday, 28 February 2019 15:47:41 GMT Rich Freeman wrote: > > > In general it is usually simplest to just remove /usr/portage > > anytime > > you change the sync settings. At least until portage gets smarter > > about it. > > That works well on a sufficiently powerful box; it only took - oh, I > don't > know - maybe a couple of minutes on this workstation. On my little > Atom box, > though, it takes 75 minutes. > > [OT] > Evidence is mounting that the Atom box is in terminal decline. I get > things > like batches of files in the portage tree changing owner, and then > when I > correct that, long lists of supposedly locally changed ebuilds > preventing > syncing. And when I boot weekly into its little rescue system to > backup the > main system, the root filesystem remounts itself read-only while tar > is > running. Smartd recognises the SSD and runs daily tests, but reports > no > errors. No amount of wiping and reinstalling has helped so far. > What filesystem are you running and how old is the SSD? That sounds like some of the symptoms EXT4 had on early generation flash media where its assumptions about what order writes would physically make it to the disk in were wrong, leading to corruption. So unless it was working correctly at some point in the past, try a different filesystem. EXT3 or BTRFS didn't have the same problems. If it's just that the SSD is failing, then get a new one before something important gets damaged and you have to redo the whole thing. The self-test capability of storage media is almost universally horrible and you generally don't get a failure report until your data has already been lost. If your SMART output gives you the raw statistics on the device instead of just pass/fail then analyzing that usually gives a better indication of whether something is about to go wrong. LMP signature.asc Description: This is a digitally signed message part
Re: [gentoo-user] Emerge --sync source
On Thursday, 28 February 2019 15:47:41 GMT Rich Freeman wrote: > In general it is usually simplest to just remove /usr/portage anytime > you change the sync settings. At least until portage gets smarter > about it. That works well on a sufficiently powerful box; it only took - oh, I don't know - maybe a couple of minutes on this workstation. On my little Atom box, though, it takes 75 minutes. [OT] Evidence is mounting that the Atom box is in terminal decline. I get things like batches of files in the portage tree changing owner, and then when I correct that, long lists of supposedly locally changed ebuilds preventing syncing. And when I boot weekly into its little rescue system to backup the main system, the root filesystem remounts itself read-only while tar is running. Smartd recognises the SSD and runs daily tests, but reports no errors. No amount of wiping and reinstalling has helped so far. -- Regards, Peter.
Re: [gentoo-user] Emerge --sync source
On Thu, Feb 28, 2019 at 10:41 AM Peter Humphrey wrote: > > On Thursday, 28 February 2019 08:43:13 GMT Davyd McColl wrote: > > > Well, that's pretty-much how git works -- that local repo was still pointing > > to the old remote. Updating your repos.conf won't change that as the old > > remote is stored in config in the .git folder. > > OK. It'd be helpful if the handbook said that, or somewhere else in the docs. > Without that, the clear impression is that repos.conf is the place to specify > the remote source. If you're going to migrate it in-place you really should set it in both places. Otherwise you'll end up with a surprise if you remove /usr/portage. In general it is usually simplest to just remove /usr/portage anytime you change the sync settings. At least until portage gets smarter about it. -- Rich
Re: [gentoo-user] Emerge --sync source
On Thursday, 28 February 2019 08:43:13 GMT Davyd McColl wrote: > > On 2019/02/28 10:36:35, Peter Humphrey wrote: > > I have a little server box on my LAN, which I use as a git server. I'm > > having a bit of trouble with it pro tem so I decided to switch the git > > sync source on this box. > > > > I removed the entry pointing to the local server in repos.conf/gentoo.conf > > and put in 'sync-uri = https://github.com/gentoo-mirror/gentoo.git' > > > > Emerge --sync still insisted on going to the local server, which was not > > there so it stopped. > > > > I had to remove /usr/portage/.git before the repos.conf/gentoo.conf entry > > was respected. And that meant stripping out the whole of /usr/portage and > > fetching the whole lot again. > Well, that's pretty-much how git works -- that local repo was still pointing > to the old remote. Updating your repos.conf won't change that as the old > remote is stored in config in the .git folder. OK. It'd be helpful if the handbook said that, or somewhere else in the docs. Without that, the clear impression is that repos.conf is the place to specify the remote source. > However, if you need to to this again, you could: 1) change repos.conf (in > case you ever wipe out /usr/portage again -- the url there is only used for > initial clone) 1) in /usr/portage, run `git remote set-url origin ` > -- this informs git of the change, and your next fetch should work as > expected. Useful tip - thanks. > I guess emerge could check this and set it for the user, but currently, it > apparently doesn't. Good idea. I hope a suitable developer is listening... -- Regards, Peter.
Re: [gentoo-user] Emerge --sync source
I filed a bug report https://bugs.gentoo.org/679040. Yes, currently you need to update your git config manually everytime you change your git remote.
Re: [gentoo-user] Emerge --sync source
On 2019/02/28 10:36:35, Peter Humphrey wrote: Hello list, I have a little server box on my LAN, which I use as a git server. I'm having a bit of trouble with it pro tem so I decided to switch the git sync source on this box. I removed the entry pointing to the local server in repos.conf/gentoo.conf and put in 'sync-uri = https://github.com/gentoo-mirror/gentoo.git' Emerge --sync still insisted on going to the local server, which was not there so it stopped. I had to remove /usr/portage/.git before the repos.conf/gentoo.conf entry was respected. And that meant stripping out the whole of /usr/portage and fetching the whole lot again. Well, that's pretty-much how git works -- that local repo was still pointing to the old remote. Updating your repos.conf won't change that as the old remote is stored in config in the .git folder. However, if you need to to this again, you could: 1) change repos.conf (in case you ever wipe out /usr/portage again -- the url there is only used for initial clone) 1) in /usr/portage, run `git remote set-url origin ` -- this informs git of the change, and your next fetch should work as expected. I guess emerge could check this and set it for the user, but currently, it apparently doesn't. Is this expected behaviour? -- Regards, Peter.
Re: [gentoo-user] Emerge --sync fails on excluded stuff
On 2018-10-22 16:26, Walter Dnes wrote: I have a 10 year old Core2 as an emergency backup machine. Let's just say it's not as fast as modern machines. To speed up "emerge --sync". I put a bunch of unneeded stuff in an "rsync_excludes" file (attached). Now "emerge --sync" has started failing, because it obviously can't verify the missing directories/files. Here's the error message... * Manifest timestamp: 2018-10-22 13:08:41 UTC * Valid OpenPGP signature found: * - primary key: DCD05B71EAB94199527F44ACDB6B8C1F96D8BF6D * - subkey: E1D6ABB63BFCFB4BA02FDF1CEC590EEAC9189250 * - timestamp: 2018-10-22 13:08:41 UTC * Verifying /usr/portage/.tmp-unverified-download-quarantine ...!!! Manifest verification failed: Manifest mismatch for app-dicts/Manifest.gz __exists__: expected: True, have: False app_dicts is the first entry in my "rsync_excludes", and I assume that all the other entries would be problematic too. How do I turn off this bleeping "helpful" verification feature? It fails at the first exclude entry always - I wrote this already. The whole manifest verification is buggy like hell - not well deliberated. Even if you sync from a local mirror it loads the key from the default key server. My suggestion is to disable it if you don't stay with the default. -- Sent with eQmail-1.11 beta - a fork of djb's famous qmail
Re: [gentoo-user] emerge --sync, where is the URL configured?
On 16:57 Wed 20 Jun, Ian Zimmerman wrote: > I am following the instructions here: > > https://wiki.gentoo.org/wiki/Chroot > > Got to the emerge --sync step, and it works! But why? There is no > $CHROOT/etc/portage/repos.conf, let alone a gentoo.conf file inside it. > So how does it know where to sync from? How about /usr/share/portage/config/repos.conf?
Re: [gentoo-user] emerge --sync, where is the URL configured?
On Wed, 20 Jun 2018 16:57:47 -0700, Ian Zimmerman wrote: > https://wiki.gentoo.org/wiki/Chroot > > Got to the emerge --sync step, and it works! But why? There is no > $CHROOT/etc/portage/repos.conf, let alone a gentoo.conf file inside it. > So how does it know where to sync from? Is there a SYNC entry in make.conf? If neither this nor a repos.conf file is present, portage uses a default. See man make.conf. -- Neil Bothwick How is it one careless match can start a forest fire, but it takes a whole box to start a campfire? pgpf_2O52i6r0.pgp Description: OpenPGP digital signature
Re: [gentoo-user] emerge --sync
Jorge Almeida wrote: > I want to do emerge --sync on computer A and then update computer B by > copying /usr/portage. Is this safe? The point is: does emerge --sync > just updates the contents of /usr/portage or does it also change > something else ? > > TIA > > Jorge Almeida > > This may be a solution. net-proxy/http-replicator I've used it here in the past when I had two rigs. Dale :-) :-)
Re: [gentoo-user] emerge --sync
Den 24. okt. 2016 17:21, skrev Jorge Almeida: > I want to do emerge --sync on computer A and then update computer B by > copying /usr/portage. Is this safe? The point is: does emerge --sync > just updates the contents of /usr/portage or does it also change > something else ? I have one box that I run emerge --sync on, and then export /usr/portage with nfs to a few other gentoo boxes. Works well. For eix, you can do eix-update on the nfs-clients. I have been experimenting with building once and using the nfs-server as binhost. Small variations between the boxes have so far made me abandon that, but a single version of /usr/portage has given me no headaches.
[gentoo-user] Re: Re: Re: Re: [Gentoo-User] emerge --sync likely to kill SSD?
Rich Freeman ri...@gentoo.org schrieb: On Sun, Jun 22, 2014 at 7:44 AM, Kai Krakow hurikha...@gmail.com wrote: I don't see where you could lose the volume management features. You just add device on top of the bcache device after you initialized the raw device with a bcache superblock and attached it. The rest works the same, just that you use bcacheX instead of sdX devices. Ah, didn't realize you could attach/remove devices to bcache later. Presumably it handles device failures gracefully, ie exposing them to the underlying filesystem so that it can properly recover? I'm not sure if multiple partitions can share the same cache device partition but more or less that's it: Initialize bcache, then attach your backing devices, then add those bcache devices to your btrfs. I don't know how errors are handled, tho. But as with every caching technique (even in ZFS) your data is likely toast if the cache device dies in the middle of action. Thus, you should put bcache on LVM RAID if you are going to use it for write caching (i.e. write-back mode). Read caching should be okay (write-through mode). Bcache is a little slower than other flash-cache implementations because it only reports data as written back to the FS if it reached stable storage (which can be the cache device, tho, if you are using write-back mode). It was also designed with unexpected reboots in mind, read. It will replay transactions from its log on reboot. This means, you can have unstable data conditions on the raw device which is why you should never try to use that directly, e.g. from a rescue disk. But since bcache wraps the partition with its own superblock this mistake should be impossible. I'm not sure how graceful device failures are handled. I suppose in write- back mode you can get into trouble because it's too late for bcache to tell the FS that there is a write error when it already confirmed that stable storage has been hit. Maybe it will just keep the data around so you could swap devices or will report the error next time when data is written to that location. It probably interferes with btrfs RAID logic on that matter. The only problem with doing stuff like this at a lower level (both write and read caching) is that it isn't RAID-aware. If you write 10GB of data, you use 20GB of cache to do it if you're mirrored, because the cache doesn't know about mirroring. Yes, it will write double the data to the cache then - but only if btrfs also did actually read both copies (which it probably does not because it has checksums and does not need to compare data, and lets just ignore the case that another process could try to read the same data from the other raid member later, that case should become optimized-out by the OS cache). Otherwise both caches should work pretty individually with their own set of data depending on how btrfs uses each device individually. Remember that btrfs raid is not a block-based raid where block locations would match 1:1 on each device. Btrfs raid can place one mirror of data in two completely different locations on each member device (which is actually a good thing in case block errors accumulate in specific locations for a faulty model of a disk). In case of write caching it will of course cache double the data (because both members will be written to). But I think that's okay for the same reasons, except it will wear your cache device faster. But in that case I suggest to use individual SSDs for each btrfs member device anyways. It's not optimal, I know. Could be useful to see some best practices and pros/cons on that topic (individual cache device per btrfs member vs. bcache on LVM RAID with bcache partitions on the RAID for all members). I think the best strategy depends on if you are write-most or read-most. Thanks for mentioning. Interesting thoughts. ;-) Offhand I'm not sure if there are any performance penalties as well around the need for barriers/etc with the cache not being able to be relied on to do the right thing in terms of what gets written out - also, the data isn't redundant while it is on the cache, unless you mirror the cache. This is partialy what I outlined above. I think in case of write-caching, there is no barriers pass-thru needed. Bcache will confirm the barriers and that's all the FS needs to know (because bcache is supervising the FS, all requests go through the bcache layer, no direct access to the backing device). Of course, it's then bcache's job to ensure everything gets written out correctly in the background (whenever it feels to do so). But it can use its own write-barriers to ensure that for the underlying device - that's nothing the FS has to care about. Performance should be faster anyway because, well, you are writing to a faster device - that is what bcache is all about, isn't it? ;-) I don't think write-barriers for read caching are needed, at least not from point of view of the FS. The
Re: [gentoo-user] Re: Re: Re: Re: [Gentoo-User] emerge --sync likely to kill SSD?
On Tue, Jun 24, 2014 at 2:34 PM, Kai Krakow hurikha...@gmail.com wrote: I'm not sure if multiple partitions can share the same cache device partition but more or less that's it: Initialize bcache, then attach your backing devices, then add those bcache devices to your btrfs. Ah, if you are stuck with one bcache partition per cached device then that will be fairly painful to manage. Yes, it will write double the data to the cache then - but only if btrfs also did actually read both copies (which it probably does not because it has checksums and does not need to compare data, and lets just ignore the case that another process could try to read the same data from the other raid member later, that case should become optimized-out by the OS cache). I didn't realize you were proposing read caching only. If you're only caching reads then obviously that is much safer. I think with btrfs in raid1 mode with only two devices you can tell it to prefer a particular device for reading in which case you could just bcache that drive. It would only read from the other drive if the cache failed. However, I don't think btrfs lets you manually arrange drives into array-like structures. It auto-balances everything which is usually a plus, but if you have 30 disks you can't tell it to treat them as 6x 5-disk RAID5s vs one 30-disk raid5 (I think). Rich
[gentoo-user] Re: Re: Re: [Gentoo-User] emerge --sync likely to kill SSD?
Rich Freeman ri...@gentoo.org schrieb: On Sat, Jun 21, 2014 at 3:24 PM, Kai Krakow hurikha...@gmail.com wrote: And while we are at it, I'd also like to mention bcache. Tho, conversion is not straight forward. However, I'm going to try that soon for my spinning rust btrfs. I contemplated that, but I'd really like to see btrfs support something more native. Bcache is way too low-level for me and strikes me as inefficient as a result. Plus, since it sits UNDER btrfs you'd probably lose all the fancy volume management features. I don't see where you could lose the volume management features. You just add device on top of the bcache device after you initialized the raw device with a bcache superblock and attached it. The rest works the same, just that you use bcacheX instead of sdX devices. Bcache is a general approach and it seems to work very well for that already. There are hot data tracking patches and proposals to support adding a cache device to the btrfs pool and let btrfs migrate data back and forth between each. That would be native. But it still would lack the advanced features ZFS implements to make use of such caching devices, implementing even different strategies for ZIL, ARC, and L2ARC. That's the gap bcache tries to jump. ZFS has ssd caching as part of the actual filesystem, and that seems MUCH cleaner. Yes, it is much more mature in that regard. Comparing with ZFS, bcache is a lot like ZIL, while hot data relocation in btrfs would be a lot like L2ARC. ARC is a special purpose RAM cache separate from the VFS caches which has special knowledge about ZFS structures to keep performance high. Some filesystems implement something similar by keeping tree structures completely in RAM. I think, both bcache and hot data tracking take parts of the work that ARC does for ZFS - note that hot data tracking is a generic VFS interface, while hot data relocation is something from btrfs. Both work together but it is not there yet. From that point of view, I don't think something like ZIL should be implemented in btrfs itself but as a generic approach like bcache so every component in Linux can make use of it. Hot data relocation OTOH is interesting from another point of view and may become part of future btrfs as it benefits from knowledge about the filesystem itself, using a generic interface like hot data tracking in VFS - so other components can make use of that, too. A ZIL-like cache and hot data relocation could probably solve a lot of fragmentation issues (especially a ZIL-like cache), so I hope work for that will get pushed a little more soon. Having to prepare devices for bcache is kind of a show-stopper because it is no drop-in component that way. But OTOH I like that approach better than dm- cache because it protects from using the backing device without going through the caching layer which could otherwise severely damage your data, and you get along with fewer devices and don't need to size a meta device (which probably needs to grow later if you add devices, I don't know). -- Replies to list only preferred.
Re: [gentoo-user] Re: Re: Re: [Gentoo-User] emerge --sync likely to kill SSD?
On Sun, Jun 22, 2014 at 7:44 AM, Kai Krakow hurikha...@gmail.com wrote: I don't see where you could lose the volume management features. You just add device on top of the bcache device after you initialized the raw device with a bcache superblock and attached it. The rest works the same, just that you use bcacheX instead of sdX devices. Ah, didn't realize you could attach/remove devices to bcache later. Presumably it handles device failures gracefully, ie exposing them to the underlying filesystem so that it can properly recover? From that point of view, I don't think something like ZIL should be implemented in btrfs itself but as a generic approach like bcache so every component in Linux can make use of it. Hot data relocation OTOH is interesting from another point of view and may become part of future btrfs as it benefits from knowledge about the filesystem itself, using a generic interface like hot data tracking in VFS - so other components can make use of that, too. The only problem with doing stuff like this at a lower level (both write and read caching) is that it isn't RAID-aware. If you write 10GB of data, you use 20GB of cache to do it if you're mirrored, because the cache doesn't know about mirroring. Offhand I'm not sure if there are any performance penalties as well around the need for barriers/etc with the cache not being able to be relied on to do the right thing in terms of what gets written out - also, the data isn't redundant while it is on the cache, unless you mirror the cache. Granted, if you're using it for write intent logging then there isn't much getting around that. Having to prepare devices for bcache is kind of a show-stopper because it is no drop-in component that way. But OTOH I like that approach better than dm- cache because it protects from using the backing device without going through the caching layer which could otherwise severely damage your data, and you get along with fewer devices and don't need to size a meta device (which probably needs to grow later if you add devices, I don't know). And this is the main thing keeping me away from it. It is REALLY painful to migrate to/from. Having it integrated into the filesystem delivers all the same benefits of not being able to mount it without the cache present. Now excuse me while I go fix my btrfs (I tried re-enabling snapper and it again got the filesystem into a worked-up state after trying to clean up half a dozen snapshots at the same time - it works fine until you go and try to write a lot of data to it, then it stops syncing though you don't necessarily notice until a few hours later when the write cache exhausts RAM and on reboot your disk reverts back a few hours). I suspect that if I just treat it gently for a few hours btrfs will clean up the mess and it will work normally again, but the damage apparently persists after a reboot if you go heavy in the disk too quickly... Rich
Re: [gentoo-user] Re: [Gentoo-User] emerge --sync likely to kill SSD?
On Friday 20 June 2014 19:48:14 Kai Krakow wrote: microcai micro...@fedoraproject.org schrieb: rsync is doing bunch of 4k ramdon IO when updateing portage tree, that will kill SSDs with much higher Write Amplification Factror. I have a 2year old SSDs that have reported Write Amplification Factor of 26. I think the only reason is that I put portage tree on this SSD to speed it up. Use a file system that turns random writes into sequential writes, like the pretty newcomer f2fs. You could try using it for your rootfs but currently I suggest just creating a separate partition for it and either mount it as /usr/portage or symlink that dir into this directory (that way you could use it for other purposes, too, that generate random short writes, like log files). Well, there's a surprise! Thanks for mentioning f2fs. I've just converted my Atom box's seven partitions to it, recompiled the kernel to include it, changed the fstab entries and rebooted. It just worked. ---8 I'd also suggest not to use the discard mount options and instead create a cronjob that runs fstrim on the SSD devices. But YMMV. I found that fstrim can't work on f2fs file systems. I don't know whether discard works yet. Thanks again. -- Regards Peter
Re: [gentoo-user] Re: [Gentoo-User] emerge --sync likely to kill SSD?
On Sat, Jun 21, 2014 at 10:27 AM, Peter Humphrey pe...@prh.myzen.co.uk wrote: I found that fstrim can't work on f2fs file systems. I don't know whether discard works yet. Fstrim is to be preferred over discard in general. However, I suspect neither is needed for something like f2fs. Being log-based it doesn't really overwrite data in place. I suspect that it waits until an entire region of the disk is unused and then it TRIMs the whole region. However, I haven't actually used it and only know the little I've read about it. That is the principle of a log-based filesystem. I'm running btrfs on my SSD root, which is supposed to be decent for flash, but the SMART attributes of my drive aren't well-documented so I couldn't tell you what the erase count is up to. Rich
[gentoo-user] Re: Re: [Gentoo-User] emerge --sync likely to kill SSD?
Rich Freeman ri...@gentoo.org schrieb: On Sat, Jun 21, 2014 at 10:27 AM, Peter Humphrey pe...@prh.myzen.co.uk wrote: I found that fstrim can't work on f2fs file systems. I don't know whether discard works yet. Fstrim is to be preferred over discard in general. However, I suspect neither is needed for something like f2fs. Being log-based it doesn't really overwrite data in place. I suspect that it waits until an entire region of the disk is unused and then it TRIMs the whole region. F2fs prefers to fill an entire erase block before touching the next. It also tries to coalese small writes into 16k blocks before submitting them to disk. And according to the docs it supports trim/discard internally. However, I haven't actually used it and only know the little I've read about it. That is the principle of a log-based filesystem. There's an article at LWN [1] and in the comments you can find a few important information about the technical details. Posted Oct 11, 2012 21:11 UTC (Thu) by arnd: | * Wear leveling usually works by having a pool of available erase blocks | in the drive. When you write to a new location, the drive takes on block | out of that pool and writes the data there. When the drive thinks you | are done writing to one block, it cleans up any partially written data | and puts a different block back into the pool. | * f2fs tries to group writes into larger operations of at least page size | (16KB or more) to be efficient, current FTLs are horribly bad at 4KB | page size writes. It also tries to fill erase blocks (multiples of 2MB) | in the order that the devices can handle. | * logfs actually works on block devices but hasn't been actively worked on | over the last few years. f2fs also promises better performance by using | only 6 erase blocks concurrently rather than 12 in the case of logfs. A | lot of the underlying principles are the same though. | * The industry is moving away from raw flash interfaces towards eMMC and | related technologies (UFS, SD, ...). We are not going back to raw flash | any time soon, which is unfortunate for a number of reasons but also has | a few significant advantages. Having the FTL take care of bad block | management and wear leveling is one such advantage, at least if they get | it right. According to wikipedia [2], some more interesting features are on the way, like compression and data deduplication to lower the impact of writes. [1]: http://lwn.net/Articles/518988/ [2]: http://en.wikipedia.org/wiki/F2FS -- Replies to list only preferred.
[gentoo-user] Re: Re: [Gentoo-User] emerge --sync likely to kill SSD?
Peter Humphrey pe...@prh.myzen.co.uk schrieb: On Friday 20 June 2014 19:48:14 Kai Krakow wrote: microcai micro...@fedoraproject.org schrieb: rsync is doing bunch of 4k ramdon IO when updateing portage tree, that will kill SSDs with much higher Write Amplification Factror. I have a 2year old SSDs that have reported Write Amplification Factor of 26. I think the only reason is that I put portage tree on this SSD to speed it up. Use a file system that turns random writes into sequential writes, like the pretty newcomer f2fs. You could try using it for your rootfs but currently I suggest just creating a separate partition for it and either mount it as /usr/portage or symlink that dir into this directory (that way you could use it for other purposes, too, that generate random short writes, like log files). Well, there's a surprise! Thanks for mentioning f2fs. I've just converted my Atom box's seven partitions to it, recompiled the kernel to include it, changed the fstab entries and rebooted. It just worked. It's said to be twice as fast with some workloads (especially write workloads). Can you confirm that? I didn't try it that much yet - usually I use it for pendrives only. I have no experience using it for rootfs. And while we are at it, I'd also like to mention bcache. Tho, conversion is not straight forward. However, I'm going to try that soon for my spinning rust btrfs. -- Replies to list only preferred.
Re: [gentoo-user] Re: Re: [Gentoo-User] emerge --sync likely to kill SSD?
On Sat, Jun 21, 2014 at 3:24 PM, Kai Krakow hurikha...@gmail.com wrote: And while we are at it, I'd also like to mention bcache. Tho, conversion is not straight forward. However, I'm going to try that soon for my spinning rust btrfs. I contemplated that, but I'd really like to see btrfs support something more native. Bcache is way too low-level for me and strikes me as inefficient as a result. Plus, since it sits UNDER btrfs you'd probably lose all the fancy volume management features. ZFS has ssd caching as part of the actual filesystem, and that seems MUCH cleaner. Rich
[gentoo-user] Re: [Gentoo-User] emerge --sync likely to kill SSD?
microcai micro...@fedoraproject.org schrieb: rsync is doing bunch of 4k ramdon IO when updateing portage tree, that will kill SSDs with much higher Write Amplification Factror. I have a 2year old SSDs that have reported Write Amplification Factor of 26. I think the only reason is that I put portage tree on this SSD to speed it up. Use a file system that turns random writes into sequential writes, like the pretty newcomer f2fs. You could try using it for your rootfs but currently I suggest just creating a separate partition for it and either mount it as /usr/portage or symlink that dir into this directory (that way you could use it for other purposes, too, that generate random short writes, like log files). Then, I'd recommend changing your scheduler to deadline, bump up the io queue depth to a much higher value (echo -n 2048 /sys/block/sdX/queue/nr_requests) and then change the dirty io flusher to not run as early as it usually would (change vm.dirty_writeback_centisecs to 1500 and vm.dirty_expire_centisecs to 3000). That way the vfs layer has a chance to better coalesce multi-block writes into one batch write, and f2fs will take care of doing it in sequential order. I'd also suggest not to use the discard mount options and instead create a cronjob that runs fstrim on the SSD devices. But YMMV. As a safety measure, only ever partition and use only 70-80% of your SSD so it can reliably do its wear-leveling. It will improve lifetime and keep the performance up even with filled filesystems. -- Replies to list only preferred.
Re: [gentoo-user] Re: [Gentoo-User] emerge --sync likely to kill SSD?
2014-06-21 1:48 GMT+08:00 Kai Krakow hurikha...@gmail.com: microcai micro...@fedoraproject.org schrieb: rsync is doing bunch of 4k ramdon IO when updateing portage tree, that will kill SSDs with much higher Write Amplification Factror. I have a 2year old SSDs that have reported Write Amplification Factor of 26. I think the only reason is that I put portage tree on this SSD to speed it up. Use a file system that turns random writes into sequential writes, like the pretty newcomer f2fs. You could try using it for your rootfs but currently I suggest just creating a separate partition for it and either mount it as /usr/portage or symlink that dir into this directory (that way you could use it for other purposes, too, that generate random short writes, like log files). Then, I'd recommend changing your scheduler to deadline, bump up the io queue depth to a much higher value (echo -n 2048 /sys/block/sdX/queue/nr_requests) and then change the dirty io flusher to not run as early as it usually would (change vm.dirty_writeback_centisecs to 1500 and vm.dirty_expire_centisecs to 3000). That way the vfs layer has a chance to better coalesce multi-block writes into one batch write, and f2fs will take care of doing it in sequential order. I'd also suggest not to use the discard mount options and instead create a cronjob that runs fstrim on the SSD devices. But YMMV. As a safety measure, only ever partition and use only 70-80% of your SSD so it can reliably do its wear-leveling. It will improve lifetime and keep the performance up even with filled filesystems. -- many thanks to all of you! no I've put my portage tree on an F2FS partation now.
Re: [gentoo-user] emerge --sync issue on only one comp on LAN
On Tuesday 01 Oct 2013 15:20:06 Bruce Hill wrote: There are 3 (or more) computers which sync (sometimes daily) on my LAN at work: server, router, and workstation. server has issues almost all the time getting a rsync server (for lack of better way to state it). All three comps have the exact same SYNC in make.conf: mingdao@server ~ $ grep SYNC /etc/make.conf SYNC=rsync://rsync.us.gentoo.org/gentoo-portage [snip ...] Can anyone see something obvious, or tell me where to begin checking or something to change? I assume that this problem occurs even when trying to sync the server alone, rather than concurrently with the other two machines. If not, then the rsync mirror may be blocking you due to repeated attempts from the same source IP address. Some things to check: Have you looked at the router's logs to see what happens to the packets? When this fault occurs, have you checked that the server can still resolve URLs and there is no DNS problem? Finally, as has already been mentioned, you could set up a LAN rsync mirror for your machines, but this does not answer why the particular machine is having connection problems. -- Regards, Mick signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] emerge --sync issue on only one comp on LAN
On Wed, Oct 2, 2013 at 4:51 PM, Alan McKinnon alan.mckin...@gmail.com wrote: On 02/10/2013 19:37, Paul Hartman wrote: On Tue, Oct 1, 2013 at 10:45 AM, Alan McKinnon alan.mckin...@gmail.com wrote: On 01/10/2013 17:17, Greg Turner wrote: Rsync mirrors don't grow on trees, man. People pay good money to provide that service to us. You should seriously be embarrassed to have posted this. Really? Then you can all use mine with the greatest of pleasure: SYNC=rsync://ftp.is.co.za/gentoo-portage I have the NetOps team BEGGING me weekly to try and generate more traffic out of our network going international. The in-out ratio on the peering links is seriously screwed and they badly want something to even it out a bit :-) Challenge accepted. :) Here's my sync summary (I'm in the USA): Number of files: 174305 Number of files transferred: 50913 ^ Total file size: 305.99M bytes Total transferred file size: 73.98M bytes Literal data: 73.98M bytes Matched data: 0 bytes File list size: 4.31M File list generation time: 343.526 seconds ^^^ File list transfer time: 0.000 seconds Total bytes sent: 1.12M Total bytes received: 41.37M sent 1.12M bytes received 41.37M bytes 64.33K bytes/sec total size is 305.99M speedup is 7.20 You don't sync very often, right? I usually sync manually daily or every other day if I'm busy and don't get a chance. I assumed there was some mass change to ebuild headers or license text or something which caused everything in the tree to get touched this week. My local portage tree is on a fast SSD in an 8-core box with 32GB of RAM and a 100mbit internet connection, so the bottleneck hopefully is not on my side of the transaction. ;) Let's do some more trials. Between yesterday and today, I have synced with my normal mirror, but I'm syncing with your server again now: Number of files: 174410 Number of files transferred: 17372 Total file size: 306.28M bytes Total transferred file size: 22.32M bytes Literal data: 22.32M bytes Matched data: 0 bytes File list size: 4.31M File list generation time: 379.920 seconds File list transfer time: 0.000 seconds Total bytes sent: 382.35K Total bytes received: 15.71M sent 382.35K bytes received 15.71M bytes 29.33K bytes/sec total size is 306.28M speedup is 19.04 Now I'm immediately doing another sync, first deleting timestamp.chk to force it to sync again. There should be zero files to transfer (except the timestamp file). Number of files: 174410 Number of files transferred: 1 Total file size: 306.28M bytes Total transferred file size: 32 bytes Literal data: 32 bytes Matched data: 0 bytes File list size: 4.31M File list generation time: 28.612 seconds File list transfer time: 0.000 seconds Total bytes sent: 183 Total bytes received: 4.31M sent 183 bytes received 4.31M bytes 128.75K bytes/sec total size is 306.28M speedup is 71.01 Now I'm switching back to my beloved mirror.steadfast.net and running another sync. Number of files: 174409 Number of files transferred: 17364 Total file size: 306.30M bytes Total transferred file size: 21.74M bytes Literal data: 21.74M bytes Matched data: 0 bytes File list size: 4.39M File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 355.23K Total bytes received: 15.67M sent 355.23K bytes received 15.67M bytes 191.93K bytes/sec total size is 306.30M speedup is 19.11 Interestingly it transferred almost the same number of files as my first sync with yours. Comparing timestamps, your server's latest update is about 5 hours older than Steadfast's, so things must be changing frequently in portage these days! 17k changes in 5 hours... My ping to your server is 300ms, my ping to steadfast is 18ms. I don't know anything about how rsync works behind the curtain, if a higher latency would cause the file list generation to be slower, or if that is a measurement of server performance or something else. Total sync times from my log: 1380814364: Starting rsync with rsync://196.4.160.12/gentoo-portage 1380814916: === Sync completed with rsync://196.4.160.12/gentoo-portage (first sync, 17k files updated, 552 seconds) 1380815150: Starting rsync with rsync://196.4.160.12/gentoo-portage 1380815188: === Sync completed with rsync://196.4.160.12/gentoo-portage (sync with no updates except timestamp.chk, 38 seconds) 1380815292: Starting rsync with rsync://208.100.4.53/gentoo-portage 1380815375: === Sync completed with rsync://208.100.4.53/gentoo-portage (re-sync with steadfast, 17k files updated, 83 seconds) 1380816062: Starting rsync with rsync://208.100.4.53/gentoo-portage 1380816074: === Sync completed with rsync://208.100.4.53/gentoo-portage (sync with no updates except timestamp.chk, 12 seconds) HTH and thanks for the mirror :) Paul
Re: [gentoo-user] emerge --sync issue on only one comp on LAN
On 03/10/2013 18:57, Paul Hartman wrote: On Wed, Oct 2, 2013 at 4:51 PM, Alan McKinnon alan.mckin...@gmail.com wrote: On 02/10/2013 19:37, Paul Hartman wrote: On Tue, Oct 1, 2013 at 10:45 AM, Alan McKinnon alan.mckin...@gmail.com wrote: On 01/10/2013 17:17, Greg Turner wrote: [snip] You don't sync very often, right? I usually sync manually daily or every other day if I'm busy and don't get a chance. I assumed there was some mass change to ebuild headers or license text or something which caused everything in the tree to get touched this week. My local portage tree is on a fast SSD in an 8-core box with 32GB of RAM and a 100mbit internet connection, so the bottleneck hopefully is not on my side of the transaction. ;) Let's do some more trials. Between yesterday and today, I have synced with my normal mirror, but I'm syncing with your server again now: Number of files: 174410 Number of files transferred: 17372 Total file size: 306.28M bytes Total transferred file size: 22.32M bytes Literal data: 22.32M bytes Matched data: 0 bytes File list size: 4.31M File list generation time: 379.920 seconds File list transfer time: 0.000 seconds Total bytes sent: 382.35K Total bytes received: 15.71M sent 382.35K bytes received 15.71M bytes 29.33K bytes/sec total size is 306.28M speedup is 19.04 Now I'm immediately doing another sync, first deleting timestamp.chk to force it to sync again. There should be zero files to transfer (except the timestamp file). Number of files: 174410 Number of files transferred: 1 Total file size: 306.28M bytes Total transferred file size: 32 bytes Literal data: 32 bytes Matched data: 0 bytes File list size: 4.31M File list generation time: 28.612 seconds File list transfer time: 0.000 seconds Total bytes sent: 183 Total bytes received: 4.31M sent 183 bytes received 4.31M bytes 128.75K bytes/sec total size is 306.28M speedup is 71.01 Now I'm switching back to my beloved mirror.steadfast.net and running another sync. Number of files: 174409 Number of files transferred: 17364 Total file size: 306.30M bytes Total transferred file size: 21.74M bytes Literal data: 21.74M bytes Matched data: 0 bytes File list size: 4.39M File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 355.23K Total bytes received: 15.67M sent 355.23K bytes received 15.67M bytes 191.93K bytes/sec total size is 306.30M speedup is 19.11 Interestingly it transferred almost the same number of files as my first sync with yours. Comparing timestamps, your server's latest update is about 5 hours older than Steadfast's, so things must be changing frequently in portage these days! 17k changes in 5 hours... My guess is the metadata. I'll have to do some checks on that mirror, IIRC it syncs every 6 hours, most likely steadfast syncs more often. One of the things about running a mirror is, it's very much set it up once and forget all about it evermore. Which is great and all, but users tend to spot problems long before the sysadmins do :-) My ping to your server is 300ms, my ping to steadfast is 18ms. I don't know anything about how rsync works behind the curtain, if a higher latency would cause the file list generation to be slower, or if that is a measurement of server performance or something else. I don't think latency is much of a factor but let me re-read some FAQs before commenting further. 300ms is totally normal from here to eu, uk and us - we're in deepest darkest Africa where hyenas prowl the streets[1] - and all traffic goes over undersea cable with *lots* of repeaters Total sync times from my log: 1380814364: Starting rsync with rsync://196.4.160.12/gentoo-portage 1380814916: === Sync completed with rsync://196.4.160.12/gentoo-portage (first sync, 17k files updated, 552 seconds) 1380815150: Starting rsync with rsync://196.4.160.12/gentoo-portage 1380815188: === Sync completed with rsync://196.4.160.12/gentoo-portage (sync with no updates except timestamp.chk, 38 seconds) 1380815292: Starting rsync with rsync://208.100.4.53/gentoo-portage 1380815375: === Sync completed with rsync://208.100.4.53/gentoo-portage (re-sync with steadfast, 17k files updated, 83 seconds) 1380816062: Starting rsync with rsync://208.100.4.53/gentoo-portage 1380816074: === Sync completed with rsync://208.100.4.53/gentoo-portage (sync with no updates except timestamp.chk, 12 seconds) HTH and thanks for the mirror :) Paul [1] Literally. True's bob, I kid you not. A 6 month old brown hyena this week wandered into the suburb where I live - it must have got separated from it's mother and walked 15 miles in the dark to get here -- Alan McKinnon alan.mckin...@gmail.com
Re: [gentoo-user] emerge --sync issue on only one comp on LAN
On Tue, Oct 1, 2013 at 10:45 AM, Alan McKinnon alan.mckin...@gmail.com wrote: On 01/10/2013 17:17, Greg Turner wrote: Rsync mirrors don't grow on trees, man. People pay good money to provide that service to us. You should seriously be embarrassed to have posted this. Really? Then you can all use mine with the greatest of pleasure: SYNC=rsync://ftp.is.co.za/gentoo-portage I have the NetOps team BEGGING me weekly to try and generate more traffic out of our network going international. The in-out ratio on the peering links is seriously screwed and they badly want something to even it out a bit :-) Challenge accepted. :) Here's my sync summary (I'm in the USA): Number of files: 174305 Number of files transferred: 50913 Total file size: 305.99M bytes Total transferred file size: 73.98M bytes Literal data: 73.98M bytes Matched data: 0 bytes File list size: 4.31M File list generation time: 343.526 seconds File list transfer time: 0.000 seconds Total bytes sent: 1.12M Total bytes received: 41.37M sent 1.12M bytes received 41.37M bytes 64.33K bytes/sec total size is 305.99M speedup is 7.20
Re: [gentoo-user] emerge --sync issue on only one comp on LAN
On Tue, Oct 01, 2013 at 05:45:13PM +0200, Alan McKinnon wrote: Then you can all use mine with the greatest of pleasure: SYNC=rsync://ftp.is.co.za/gentoo-portage I have the NetOps team BEGGING me weekly to try and generate more traffic out of our network going international. The in-out ratio on the peering links is seriously screwed and they badly want something to even it out a bit :-) Just for giggles I tried it too, from the US. Slower than turtles mating. ;) -- Happy Penguin Computers ') 126 Fenco Drive ( \ Tupelo, MS 38801 ^^ supp...@happypenguincomputers.com 662-269-2706 662-205-6424 http://happypenguincomputers.com/ A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing in e-mail? Don't top-post: http://en.wikipedia.org/wiki/Top_post#Top-posting
Re: [gentoo-user] emerge --sync issue on only one comp on LAN
On 02/10/2013 19:37, Paul Hartman wrote: On Tue, Oct 1, 2013 at 10:45 AM, Alan McKinnon alan.mckin...@gmail.com wrote: On 01/10/2013 17:17, Greg Turner wrote: Rsync mirrors don't grow on trees, man. People pay good money to provide that service to us. You should seriously be embarrassed to have posted this. Really? Then you can all use mine with the greatest of pleasure: SYNC=rsync://ftp.is.co.za/gentoo-portage I have the NetOps team BEGGING me weekly to try and generate more traffic out of our network going international. The in-out ratio on the peering links is seriously screwed and they badly want something to even it out a bit :-) Challenge accepted. :) Here's my sync summary (I'm in the USA): Number of files: 174305 Number of files transferred: 50913 ^ Total file size: 305.99M bytes Total transferred file size: 73.98M bytes Literal data: 73.98M bytes Matched data: 0 bytes File list size: 4.31M File list generation time: 343.526 seconds ^^^ File list transfer time: 0.000 seconds Total bytes sent: 1.12M Total bytes received: 41.37M sent 1.12M bytes received 41.37M bytes 64.33K bytes/sec total size is 305.99M speedup is 7.20 You don't sync very often, right? -- Alan McKinnon alan.mckin...@gmail.com
Re: [gentoo-user] emerge --sync issue on only one comp on LAN
On 02/10/2013 20:48, Bruce Hill wrote: On Tue, Oct 01, 2013 at 05:45:13PM +0200, Alan McKinnon wrote: Then you can all use mine with the greatest of pleasure: SYNC=rsync://ftp.is.co.za/gentoo-portage I have the NetOps team BEGGING me weekly to try and generate more traffic out of our network going international. The in-out ratio on the peering links is seriously screwed and they badly want something to even it out a bit :-) Just for giggles I tried it too, from the US. Slower than turtles mating. ;) Give the poor thing a break - it's not dedicated to Gentoo :-) There's about 11TB of mirrors on that host = full copies of every major distro, most of the popular middle teir ones and full copies of all 3 BSDs The general public keep it busy - I blame them :-) -- Alan McKinnon alan.mckin...@gmail.com
Re: [gentoo-user] emerge --sync issue on only one comp on LAN
On Wed, Oct 02, 2013 at 11:54:30PM +0200, Alan McKinnon wrote: On 02/10/2013 20:48, Bruce Hill wrote: On Tue, Oct 01, 2013 at 05:45:13PM +0200, Alan McKinnon wrote: Then you can all use mine with the greatest of pleasure: SYNC=rsync://ftp.is.co.za/gentoo-portage I have the NetOps team BEGGING me weekly to try and generate more traffic out of our network going international. The in-out ratio on the peering links is seriously screwed and they badly want something to even it out a bit :-) Just for giggles I tried it too, from the US. Slower than turtles mating. ;) Give the poor thing a break - it's not dedicated to Gentoo :-) There's about 11TB of mirrors on that host = full copies of every major distro, most of the popular middle teir ones and full copies of all 3 BSDs The general public keep it busy - I blame them :-) I didn't mean anything disrespectful (notice my winky), just that from the US it's rather slow. But it's still listed, commented, in make.conf. Thanks, Alan! -- Happy Penguin Computers ') 126 Fenco Drive ( \ Tupelo, MS 38801 ^^ supp...@happypenguincomputers.com 662-269-2706 662-205-6424 http://happypenguincomputers.com/ A: Because it messes up the order in which people normally read text. Q: Why is top-posting such a bad thing? A: Top-posting. Q: What is the most annoying thing in e-mail? Don't top-post: http://en.wikipedia.org/wiki/Top_post#Top-posting
Re: [gentoo-user] emerge --sync issue on only one comp on LAN
On 01/10/2013 16:20, Bruce Hill wrote: There are 3 (or more) computers which sync (sometimes daily) on my LAN at work: server, router, and workstation. server has issues almost all the time getting a rsync server (for lack of better way to state it). All three comps have the exact same SYNC in make.conf: mingdao@server ~ $ grep SYNC /etc/make.conf SYNC=rsync://rsync.us.gentoo.org/gentoo-portage mingdao@router ~ $ grep SYNC /etc/portage/make.conf SYNC=rsync://rsync.us.gentoo.org/gentoo-portage mingdao@workstation ~ $ grep SYNC /etc/portage/make.conf SYNC=rsync://rsync.us.gentoo.org/gentoo-portage Most likely obvious cause: You have outbound firewall rules on your server/router affecting the server. When I first noticed, it was that with the same command (alias ud) run on each computer, server *always* took a good deal longer than workstation and router. Here's the output from the emerge --sync portion of my ud alias today: alias ud='eix-sync emerge -aDjuv --changed-use @world dispatch-conf emerge -a --depclean revdep-rebuild -i clear exit' server ~ # ud * Running emerge --sync Synchronization of repository 'gentoo' located in '/usr/portage'... Starting rsync with rsync://128.61.111.9/gentoo-portage... Checking server timestamp ... timed out rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(549) [Receiver=3.0.9] Retrying... Starting retry 1 of 9 with rsync://209.59.138.21/gentoo-portage Checking server timestamp ... timed out rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(549) [Receiver=3.0.9] Retrying... Starting retry 2 of 9 with rsync://128.175.60.112/gentoo-portage Checking server timestamp ... timed out rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(549) [Receiver=3.0.9] Retrying... Starting retry 3 of 9 with rsync://129.21.171.98/gentoo-portage Checking server timestamp ... receiving incremental file list timestamp.chk Number of files: 1 Number of files transferred: 1 Total file size: 32 bytes Total transferred file size: 32 bytes Literal data: 32 bytes Matched data: 0 bytes File list size: 27 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 98 Total bytes received: 140 sent 98 bytes received 140 bytes 158.67 bytes/sec total size is 32 speedup is 0.13 receiving incremental file list router ~ # ud * Running emerge --sync Synchronization of repository 'gentoo' located in '/usr/portage'... Starting rsync with rsync://208.100.4.53/gentoo-portage... Checking server timestamp ... Welcome! This is a gentoo-portage and CentOS mirror, hosted by Steadfast Networks! http://steadfast.net Hostname: mirror.steadfast.net rsync11.us.gentoo.org IP Addresses: 208.100.4.53 2607:f128:1:3::2 Location: Chicago, IL, US Bandwidth:1000 Mbps Hardware: Dual Opteron 2212, 8 GB RAM User Limit: 40 If you experience any trouble with this mirror, please contact mir...@steadfast.net. receiving incremental file list timestamp.chk Number of files: 1 Number of files transferred: 1 Total file size: 32 bytes Total transferred file size: 32 bytes Literal data: 32 bytes Matched data: 0 bytes File list size: 27 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 98 Total bytes received: 570 sent 98 bytes received 570 bytes 445.33 bytes/sec total size is 32 speedup is 0.05 Welcome! This is a gentoo-portage and CentOS mirror, hosted by Steadfast Networks! http://steadfast.net Hostname: mirror.steadfast.net rsync11.us.gentoo.org IP Addresses: 208.100.4.53 2607:f128:1:3::2 Location: Chicago, IL, US Bandwidth:1000 Mbps Hardware: Dual Opteron 2212, 8 GB RAM User Limit: 40 If you experience any trouble with this mirror, please contact mir...@steadfast.net. receiving incremental file list workstation ~ # ud * Running emerge --sync Synchronization of repository 'gentoo' located in '/usr/portage'... Starting rsync with rsync://209.221.142.124/gentoo-portage... Checking server timestamp ... Gentoo Portage/CPAN rsync mirror Server: gentoo.llarian.net cpan.llarian.net IP(s): 209.221.142.124 2001:5d8:11::13 Bandwidth: 1000Mbps via multiple carriers User Limit: 30 Location: Seattle, WA, USA Admin Contact: Dylan Vanderhoof llar...@llarian.net receiving incremental file list timestamp.chk Number of files: 1 Number of files transferred: 1 Total file size: 32 bytes Total transferred file size: 32 bytes Literal data: 32 bytes Matched data: 0 bytes File list size: 27 File list generation time: 0.001 seconds File list transfer time: 0.000 seconds Total bytes sent: 98 Total bytes received: 392 sent 98 bytes received 392 bytes 326.67 bytes/sec total size is 32 speedup is
Re: [gentoo-user] emerge --sync issue on only one comp on LAN
On Tue, Oct 1, 2013 at 7:20 AM, Bruce Hill da...@happypenguincomputers.com wrote: There are 3 (or more) computers which sync (sometimes daily) on my LAN at work: server, router, and workstation. server has issues almost all the time getting a rsync server (for lack of better way to state it). All three comps have the exact same SYNC in make.conf: Seriously? Your problem is that an incredible build-up of bad karma has polluted your network. You are selfishly and pointlessly wasting the rsync.us.gentoo.org mirror network's resources, and your own bandwidth as well. Run rsyncd somewhere. Sync the other two systems to it. If the server has problems with outbound connectivity, use the router, I guess. Rsync mirrors don't grow on trees, man. People pay good money to provide that service to us. You should seriously be embarrassed to have posted this. -gmt
Re: [gentoo-user] emerge --sync issue on only one comp on LAN
On 01/10/2013 17:17, Greg Turner wrote: On Tue, Oct 1, 2013 at 7:20 AM, Bruce Hill da...@happypenguincomputers.com wrote: There are 3 (or more) computers which sync (sometimes daily) on my LAN at work: server, router, and workstation. server has issues almost all the time getting a rsync server (for lack of better way to state it). All three comps have the exact same SYNC in make.conf: Seriously? Your problem is that an incredible build-up of bad karma has polluted your network. You are selfishly and pointlessly wasting the rsync.us.gentoo.org mirror network's resources, and your own bandwidth as well. Run rsyncd somewhere. Sync the other two systems to it. If the server has problems with outbound connectivity, use the router, I guess. Rsync mirrors don't grow on trees, man. People pay good money to provide that service to us. You should seriously be embarrassed to have posted this. Really? Then you can all use mine with the greatest of pleasure: SYNC=rsync://ftp.is.co.za/gentoo-portage I have the NetOps team BEGGING me weekly to try and generate more traffic out of our network going international. The in-out ratio on the peering links is seriously screwed and they badly want something to even it out a bit :-) -- Alan McKinnon alan.mckin...@gmail.com
Re: [gentoo-user] emerge --sync
Silvio Siefke writes: i try to install a Gentoo Vserver by Hosteurope. Im have take the last stage archive, because the vserver Archiv is old i think. When i want run emerge --sync it gives only this message: [...] ERROR: out of memory in flist_expand [receiver] rsync error: error allocating core memory buffers (code 22) at util.c(117) [receiver=3.0.9] rsync: connection unexpectedly closed (2861 bytes received so far) [generator] rsync error: error in rsync protocol data stream (code 12) at io.c(605) [generator=3.0.9] Retrying... Can me someone tell what is it? As it says, you're out of memory. It seems you are low on RAM, what does free -m say? Maybe you need to add some swap space? Wonko
Re: [gentoo-user] emerge --sync
On Mon, 15 Oct 2012 19:50:07 +0200 Alex Schuster wo...@wonkology.org wrote: As it says, you're out of memory. It seems you are low on RAM, what does free -m say? Maybe you need to add some swap space? lvps5-35-240-192 / # free -m total used free sharedbuffers cached Mem:164600 11 164589 0 0 0 -/+ buffers/cache: 11 164589 Swap:0 0 0 Regards Silvio
Re: [gentoo-user] emerge --sync
On Mon, 15 Oct 2012 21:03:59 +0200 Silvio Siefke siefke_lis...@web.de wrote: On Mon, 15 Oct 2012 19:50:07 +0200 Alex Schuster wo...@wonkology.org wrote: As it says, you're out of memory. It seems you are low on RAM, what does free -m say? Maybe you need to add some swap space? lvps5-35-240-192 / # free -m total used free sharedbuffers cached Mem:164600 11 164589 0 0 0 -/+ buffers/cache: 11 164589 Swap:0 0 0 You have 164M of RAM, that is not enough. Packages like gcc and glibc will probably just not compile with so little RAM[1], and there is no way on a Gentoo machine to avoid compiling those. You have several options: - use something else, not Gentoo - Buy more RAM from the virtual machine provider - build on another machine and emerge the binary packages. The first is the one with the least pain. [1] I have regular 32bit x86 VMs in my test lab that struggle to properly compile big packages with 256M and sometimes even 512M is not enough - gcc is the usual culprit. -- Alan McKinnon alan.mckin...@gmail.com
Re: [gentoo-user] emerge --sync
Hello, On Mon, 15 Oct 2012 21:56:07 +0200 You have 164M of RAM, that is not enough. Packages like gcc and glibc will probably just not compile with so little RAM[1], and there is no way on a Gentoo machine to avoid compiling those. You have several options: In the description stand 1 GB, dynamic 2 GB. When thats not enough, then i not know. http://www.hosteurope.de/produkt/Virtual-Server-Linux-L For DNS Backup should enough, and when u run one time emerge package you want never have other. And Debian with Plesk what is origin on maschines is not happy life. I write to support and we see what they say. Regards Silvio
Re: [gentoo-user] emerge --sync
Alan McKinnon writes: On Mon, 15 Oct 2012 21:03:59 +0200 Silvio Siefke siefke_lis...@web.de wrote: On Mon, 15 Oct 2012 19:50:07 +0200 Alex Schuster wo...@wonkology.org wrote: As it says, you're out of memory. It seems you are low on RAM, what does free -m say? Maybe you need to add some swap space? lvps5-35-240-192 / # free -m total used free sharedbuffers cached Mem:164600 11 164589 0 0 0 -/+ buffers/cache: 11 164589 Swap:0 0 0 You have 164M of RAM, that is not enough. Packages like gcc and glibc free -m outputs megabytes, so this would mean he has 164 G of RAM, with only 11 M being used... something is wrong here. Not sure what. Wonko
Re: [gentoo-user] emerge --sync
Am 15.10.2012 22:57, schrieb Alex Schuster: Alan McKinnon writes: On Mon, 15 Oct 2012 21:03:59 +0200 Silvio Siefke siefke_lis...@web.de wrote: On Mon, 15 Oct 2012 19:50:07 +0200 Alex Schuster wo...@wonkology.org wrote: As it says, you're out of memory. It seems you are low on RAM, what does free -m say? Maybe you need to add some swap space? lvps5-35-240-192 / # free -m total used free sharedbuffers cached Mem:164600 11 164589 0 0 0 -/+ buffers/cache: 11 164589 Swap:0 0 0 You have 164M of RAM, that is not enough. Packages like gcc and glibc free -m outputs megabytes, so this would mean he has 164 G of RAM, with only 11 M being used... something is wrong here. Not sure what. Wonko You cannot trust `free` on a vserver. Just because the system has that much RAM, doesn't mean its allocated to your instance. It should still be enough to use gentoo on it. In fact, I'm doing just that. Regards, Florian Philipp signature.asc Description: OpenPGP digital signature
Re: [gentoo-user] emerge --sync
On Mon, 15 Oct 2012 22:57:17 +0200 Alex Schuster wo...@wonkology.org wrote: Alan McKinnon writes: On Mon, 15 Oct 2012 21:03:59 +0200 Silvio Siefke siefke_lis...@web.de wrote: On Mon, 15 Oct 2012 19:50:07 +0200 Alex Schuster wo...@wonkology.org wrote: As it says, you're out of memory. It seems you are low on RAM, what does free -m say? Maybe you need to add some swap space? lvps5-35-240-192 / # free -m total used free sharedbuffers cached Mem:164600 11 164589 0 0 0 -/+ buffers/cache: 11 164589 Swap:0 0 0 You have 164M of RAM, that is not enough. Packages like gcc and glibc free -m outputs megabytes, so this would mean he has 164 G of RAM, with only 11 M being used... something is wrong here. Not sure what. Oops, indeed. I read the man page (really, I did...) and my brain performed a s/mega/kilo/ -- Alan McKinnon alan.mckin...@gmail.com
Re: [gentoo-user] emerge --sync failed
On Mon, 2011-10-24 at 01:40 +0200, Hartmut Figge wrote: Greetings, emerge --sync was successfull till 'Performing Global Updates' which leaded to an error. After waiting for perhaps 1 to 2 hours i tried another 'emerge --sync' in the hope that would fix the issue. No luck. --- i5 hafi # emerge --sync Performing Global Updates: (Could take a couple of minutes if you have a lot of binary packages.) .='update pass' *='binary update' #='/var/db update' @='/var/db move' s='/var/db SLOT move' %='binary move' S='binary SLOT move' p='update /etc/portage/package.*' /usr/portage/profiles/updates/3Q-2011.. /usr/portage/profiles/updates/4Q-2011.. ERROR: Malformed update entry 'move dev-php5/dev-php5/pecl-ssh2 dev-php/dev-php5/pecl-ssh2' Traceback (most recent call last): File /usr/bin/emerge, line 43, in module retval = emerge_main() File /usr/lib/portage/pym/_emerge/main.py, line 1531, in emerge_main _global_updates(trees, mtimedb[updates], quiet=(--quiet in myopts)): File /usr/lib/portage/pym/portage/_global_updates.py, line 160, in _global_updates moves = vardb.move_ent(update_cmd, repo_match=repo_match) File /usr/lib/portage/pym/portage/dbapi/vartree.py, line 300, in move_ent origmatches = self.match(origcp, use_cache=0) File /usr/lib/portage/pym/portage/dbapi/vartree.py, line 474, in match origdep, mydb=self, use_cache=use_cache, settings=self.settings) File /usr/lib/portage/pym/portage/dbapi/dep_expand.py, line 33, in dep_expand mydep = Atom(mydep, allow_repo=True) File /usr/lib/portage/pym/portage/dep/__init__.py, line 1097, in __init__ raise InvalidAtom(self) InvalidAtom: dev-php5/dev-php5/pecl-ssh2 --- I'm seeing this chatter now. It looks like someone made a typo. And some mirrors are still picking it up. You should be able to delete /usr/portage/profiles/updates/4Q-2011 (or edit the file and fix it... it's pretty straighforward) and continue from there. -a
Re: [gentoo-user] emerge --sync kde-base/
2009/1/28 Norberto Bensa nbe...@gmail.com Hello, I'm wondering. Is it posible to emerge --sync part of the tree. For example, I want to only sync kde-base/ and kde-misc/? Thanks in advance, Norberto Have a look in make.conf.example, i believe there are some options to filter the tree, there may have been something on the forums too that helped trim the tree down aswell so theres no harm in searching there. - Nick
Re: [gentoo-user] emerge --sync kde-base/
Norberto Bensa wrote: Hello, I'm wondering. Is it posible to emerge --sync part of the tree. For example, I want to only sync kde-base/ and kde-misc/? Thanks in advance, Norberto No, there isn't. You wouldn't want to try either - there will be relevant files in other parts of the tree (eg. eclasses, related packages in other categories) that you'll also want to make sure are up-to-date. AllenJB
Re: [gentoo-user] emerge --sync kde-base/
2009/1/28 AllenJB gentoo-li...@allenjb.me.uk Norberto Bensa wrote: Hello, I'm wondering. Is it posible to emerge --sync part of the tree. For example, I want to only sync kde-base/ and kde-misc/? Thanks in advance, Norberto No, there isn't. You wouldn't want to try either - there will be relevant files in other parts of the tree (eg. eclasses, related packages in other categories) that you'll also want to make sure are up-to-date. AllenJB Agreed, i certainly wouldnt recommend it, if its a space issue your better off looking at compressing the tree, or if its speed then mounting it in RAM (or some other method, theres plenty of guides on the forums for just this). If you *really* want to, then the best (and perhaps only?) way is to use the make.conf option to pass options to rsync and then use this to tell rsync to exclude certain directories, although do note that if you try this it is totally unsupported and will likely get you flamed/ignored/laughed at if you encounter problems :) - Nick
Re: [gentoo-user] emerge --sync fails with CacheCorruption
File /usr/lib/portage/pym/cache/util.py, line 31, in mirror_cache try:entry = src_cache[x] File /usr/lib/portage/pym/cache/metadata.py, line 32, in __getitem__ return flat_hash.database.__getitem__(self, cpv) File /usr/lib/portage/pym/cache/flat_hash.py, line 38, in __getitem__ raise cache_errors.CacheCorruption(cpv, e) cache.cache_errors.CacheCorruption: dev-games/ode-0.8 is corrupt: [Errno 5] Input/output error http://bugs.gentoo.org/show_bug.cgi?id=196680 It is a bug in portage, but it's also abnormal to receive an Input/output error in that case, so you should probably run fsck on that filesystem. Fixed, thanks Zac! - Grant -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] emerge --sync fails with CacheCorruption
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Grant wrote: File /usr/lib/portage/pym/cache/util.py, line 31, in mirror_cache try:entry = src_cache[x] File /usr/lib/portage/pym/cache/metadata.py, line 32, in __getitem__ return flat_hash.database.__getitem__(self, cpv) File /usr/lib/portage/pym/cache/flat_hash.py, line 38, in __getitem__ raise cache_errors.CacheCorruption(cpv, e) cache.cache_errors.CacheCorruption: dev-games/ode-0.8 is corrupt: [Errno 5] Input/output error http://bugs.gentoo.org/show_bug.cgi?id=196680 It is a bug in portage, but it's also abnormal to receive an Input/output error in that case, so you should probably run fsck on that filesystem. Zac -BEGIN PGP SIGNATURE- Version: GnuPG v2.0.7 (GNU/Linux) iD8DBQFHHU7//ejvha5XGaMRAncHAKDOKb1mCZRZwppLJDrzUyGAf3wmFACg2COt 6A3aSb7i3ZWOdGgj/slVk0Y= =vZAK -END PGP SIGNATURE- -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] emerge --sync stopped working
Walter Dnes wrote: On Mon, Sep 17, 2007 at 01:35:24PM +0100, Neil Bothwick wrote Did you change ISPs when moving? Does your new ISP block rsync traffic? Bingo. Same ISP. Things worked OK after the move for about a month after the move. Since about Sept 8th, however, I haven't been able to send out email via my broadband ISP. I've had to fall back to my dialup ISP, which I deliberately use a different provider for. Their name, and website, is 295.ca so guess how much they charge per month. Anyhow, I just ran a successful emerge --sync via dialup. I have some complaining to do. This may not be the problem in your case but when I ran into that problem on a second machine, my mobo was bad. I suspect it had something to do with the PCI bus on mine. I also found some bad ram on there too which makes me think maybe a surge hit it or something. In my case, I tried several different ethernet cards with no help. Just a thought. Dale :-) :-) :-)
Re: [gentoo-user] emerge --sync stopped working
On Mon, 17 Sep 2007 07:32:18 -0400, Walter Dnes wrote: I had just moved, and was busy setting up my new condo, [snip] It was a major struggle. emerge --sync stopped working. I manually selected a different rsync server, and got the same error messages. I gave up and used webrsync. Did you change ISPs when moving? Does your new ISP block rsync traffic? -- Neil Bothwick deja noo - reminds you of the last time you visited Scotland signature.asc Description: PGP signature
Re: [gentoo-user] emerge --sync stopped working
On Monday 17 September 2007 13:32:18 Walter Dnes wrote: and the the same errors with other servers. I also tried downgrading from 2.6.3-r3 to -r2 and -r1, and got the same errors. Any ideas? [m3000][root][~] emerge --sync Starting rsync with rsync://134.153.48.2/gentoo-portage... Checking server timestamp ... timed out rsync error: received SIGINT, SIGTERM, or SIGHUP (code 20) at rsync.c(244) [receiver=2.6.9] Tried upgrading to a version in portage (2.6.9-r1 2.6.9-r2 2.6.9-r3)? Could be a version incompability issue. -- Naga -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] emerge --sync stopped working
On Mon, Sep 17, 2007 at 03:00:40PM +0200, Naga wrote On Monday 17 September 2007 13:32:18 Walter Dnes wrote: and the the same errors with other servers. I also tried downgrading from 2.6.3-r3 to -r2 and -r1, and got the same errors. Any ideas? Tried upgrading to a version in portage (2.6.9-r1 2.6.9-r2 2.6.9-r3)? Could be a version incompability issue. Sorry about that typo. That was supposed to read 2.6.9, and I tried -r1, -r2, and -r3. -- Walter Dnes [EMAIL PROTECTED] In linux /sbin/init is Job #1 Q. Mr. Ghandi, what do you think of Microsoft security? A. I think it would be a good idea. -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] emerge --sync stopped working
On Mon, Sep 17, 2007 at 01:35:24PM +0100, Neil Bothwick wrote Did you change ISPs when moving? Does your new ISP block rsync traffic? Bingo. Same ISP. Things worked OK after the move for about a month after the move. Since about Sept 8th, however, I haven't been able to send out email via my broadband ISP. I've had to fall back to my dialup ISP, which I deliberately use a different provider for. Their name, and website, is 295.ca so guess how much they charge per month. Anyhow, I just ran a successful emerge --sync via dialup. I have some complaining to do. -- Walter Dnes [EMAIL PROTECTED] In linux /sbin/init is Job #1 Q. Mr. Ghandi, what do you think of Microsoft security? A. I think it would be a good idea. -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] emerge --sync behind a firewall
Marko Kocić schrieb: Hi all, I'm trying to use emerge --sync behind a company firewall. I don't have a direct internet connection, only through http and socks proxies. Is there a way to configure emerge so that it can use either one of them. Thanks, Marko # emerge-webrsync -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] emerge --sync behind a firewall
maybe you should try emerge-webrsync... :/ and in the make.conf add: http_proxy=host:port O/H Marko Kocić έγραψε: Hi all, I'm trying to use emerge --sync behind a company firewall. I don't have a direct internet connection, only through http and socks proxies. Is there a way to configure emerge so that it can use either one of them. Thanks, Marko -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] emerge --sync behind a firewall
I just got answer on IRC #gentoo channel. I had to install tsocks package, which let me run through socks proxy any program. Setting http_proxy and RSYNC_PROXY didn't do the trick. On 4/19/07, Stratos Psomadakis [EMAIL PROTECTED] wrote: maybe you should try emerge-webrsync... :/ and in the make.conf add: http_proxy=host:port O/H Marko Kocić έγραψε: Hi all, I'm trying to use emerge --sync behind a company firewall. I don't have a direct internet connection, only through http and socks proxies. Is there a way to configure emerge so that it can use either one of them. Thanks, Marko -- [EMAIL PROTECTED] mailing list z� ��(��j)b� b�
Re: [gentoo-user] emerge --sync behind a firewall
On 20:02 Thu 19 Apr , Marko Kocić wrote: I just got answer on IRC #gentoo channel. I had to install tsocks package, which let me run through socks proxy any program. Setting http_proxy and RSYNC_PROXY didn't do the trick. 'net-misc/proxychains' is better than tsocks as it can force connections through socks proxy as well as http proxy (tsocks can not do http proxy). Other option is to use emerge-webrsync, which obeys $http_proxy env variable. -- Almost anything derogatory you could say about today's software design would be accurate. -- K.E. Iverson -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] Emerge --sync doesn't work anymore
On Saturday 14 April 2007 11:54:42 Benjamin Graf wrote: I added some ebuilds in /usr/local/portage and ran the ebuild foo.ebuild digest command for every ebuild I added. Now, emerge --sync gives me an error : calypso ~ # emerge --sync Starting rsync with rsync://134.68.220.74/gentoo-portage... Checking server timestamp ... rsync: --filter=H_**/files/digest-*: unknown option rsync error: syntax or usage error (code 1) at main.c(1013) !!! Rsync has reported that there is a syntax error. Please ensure !!! that your SYNC statement is proper. !!! SYNC=rsync://rsync.gentoo.org/gentoo-portage I can update my portage tree with emerge-webrsync. You can remove the --filter option from PORTAGE_RSYNC_OPTS in /etc/make.globals to make it work now. Note that it'll be added again the next time you emerge portage. Before then you should upgrade rsync. -- Bo Andresen signature.asc Description: This is a digitally signed message part.
Re: [gentoo-user] Emerge --sync doesn't work anymore
Thanks a lot ! it works. 2007/4/14, Bo Ørsted Andresen [EMAIL PROTECTED]: On Saturday 14 April 2007 11:54:42 Benjamin Graf wrote: I added some ebuilds in /usr/local/portage and ran the ebuild foo.ebuild digest command for every ebuild I added. Now, emerge --sync gives me an error : calypso ~ # emerge --sync Starting rsync with rsync://134.68.220.74/gentoo-portage... Checking server timestamp ... rsync: --filter=H_**/files/digest-*: unknown option rsync error: syntax or usage error (code 1) at main.c(1013) !!! Rsync has reported that there is a syntax error. Please ensure !!! that your SYNC statement is proper. !!! SYNC=rsync://rsync.gentoo.org/gentoo-portage I can update my portage tree with emerge-webrsync. You can remove the --filter option from PORTAGE_RSYNC_OPTS in /etc/make.globals to make it work now. Note that it'll be added again the next time you emerge portage. Before then you should upgrade rsync. -- Bo Andresen -- [EMAIL PROTECTED] mailing list
Re: [gentoo-user] emerge --sync fails
On Saturday 17 February 2007 17:45:41 Erik wrote: emerge --sync failed on one system today. I tried twice. Here is the end [SNIP] cache.cache_errors.CacheCorruption: app-accessibility/SphinxTrain-0.9.1-r1 is corrupt: dictionary update sequence element #0 has length 1; 2 is required http://bugs.gentoo.org/show_bug.cgi?id=156374 -- Bo Andresen pgpsFrjZWQggU.pgp Description: PGP signature
RE: [gentoo-user] emerge --sync connecting to 1.0.0.0 [OT, maybe]
Hey folks I've seen that exact problem with many simple routers like the one you have and various Linux distros, almost always some of the programs do work with dns server being the router itself and some don't. Namely: Web browsers and IM apps mostly work, rsync,svn git and some telnet and ftp apps mostly don't. In MS Windows almost always all work just fine. The solution for me was to find out in the router itself what are the dns servers that it uses and configure them in the system, either in resolv.conf or in conf.d/net . Hope it helps. Indeed, it works for me as well :-) Thank you Marco Yes, adding DNS servers manually to /etc/resolv.conf worked for me as well. Things kept trying to resolve to 1.0.0.0. For reference I was using a DLink 604 ADSL Modem Router (if memory serves). David -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] emerge --sync connecting to 1.0.0.0 [OT, maybe]
On 12/17/06, Mark M [EMAIL PROTECTED] wrote: On 12/16/06, marco restelli [EMAIL PROTECTED] wrote: On 12/16/06, Boyd Stephen Smith Jr. [EMAIL PROTECTED] wrote: On Saturday 16 December 2006 08:38, marco restelli [EMAIL PROTECTED] wrote about 'Re: [gentoo-user] emerge --sync connecting to 1.0.0.0 [OT, maybe]': On 12/16/06, Boyd Stephen Smith Jr. [EMAIL PROTECTED] wrote: Probably some bad DNS server... Could you post the contents of /etc/resolv.conf when your laptop is connected to the Netgear? Here it is: cat /etc/resolv.conf # Generated by dhcpcd for interface wlan0 nameserver 192.168.1.1 Actually, 192.168.1.1 is the router, which is connected to the internet with a Netgear modem, IP 192.168.0.1 Alright. This means that (most likely) your laptop is asking your router to resolve rsync.gentoo.org and that router is giving back bad response(s), at least at first. You should check the configuration of the DNS server on the router. Ok, this is already a useful information If you need further assistance, I'll need to know more about your router, specifically it's firm- and software. Also, if the router isn't running Gentoo, we may need to take this off-list of (at least) mark further messages as off-topic. It is Netgear WGR614 v6, and it is not running Gentoo. If you want to confirm the problem is with the router, break out your generic DNS clients (dig/nslookup) and network monitoring tools (tcpdump et al.). Of course, your router my simply be giving bad responses because it's getting bad responses from further upstream. Thanks for all the advices. Marco -- gentoo-user@gentoo.org mailing list I've seen that exact problem with many simple routers like the one you have and various Linux distros, almost always some of the programs do work with dns server being the router itself and some don't. Namely: Web browsers and IM apps mostly work, rsync,svn git and some telnet and ftp apps mostly don't. In MS Windows almost always all work just fine. The solution for me was to find out in the router itself what are the dns servers that it uses and configure them in the system, either in resolv.conf or in conf.d/net . Hope it helps. Indeed, it works for me as well :-) Thank you Marco -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] emerge --sync connecting to 1.0.0.0 [OT, maybe]
On Saturday 16 December 2006 06:37, marco restelli [EMAIL PROTECTED] wrote about '[gentoo-user] emerge --sync connecting to 1.0.0.0 [OT, maybe]': I am even unsure if the problem is the router or the laptop configuration or both, hence the OT. Any suggestion? Probably some bad DNS server... Could you post the contents of /etc/resolv.conf when your laptop is connected to the Netgear? -- If there's one thing we've established over the years, it's that the vast majority of our users don't have the slightest clue what's best for them in terms of package stability. -- Gentoo Developer Ciaran McCreesh pgpxfvS27hPE2.pgp Description: PGP signature
Re: [gentoo-user] emerge --sync connecting to 1.0.0.0 [OT, maybe]
On 12/16/06, Boyd Stephen Smith Jr. [EMAIL PROTECTED] wrote: On Saturday 16 December 2006 06:37, marco restelli [EMAIL PROTECTED] wrote about '[gentoo-user] emerge --sync connecting to 1.0.0.0 [OT, maybe]': I am even unsure if the problem is the router or the laptop configuration or both, hence the OT. Any suggestion? Probably some bad DNS server... Could you post the contents of /etc/resolv.conf when your laptop is connected to the Netgear? Here it is: cat /etc/resolv.conf # Generated by dhcpcd for interface wlan0 nameserver 192.168.1.1 Actually, 192.168.1.1 is the router, which is connected to the internet with a Netgear modem, IP 192.168.0.1 Marco -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] emerge --sync connecting to 1.0.0.0 [OT, maybe]
On Saturday 16 December 2006 08:38, marco restelli [EMAIL PROTECTED] wrote about 'Re: [gentoo-user] emerge --sync connecting to 1.0.0.0 [OT, maybe]': On 12/16/06, Boyd Stephen Smith Jr. [EMAIL PROTECTED] wrote: Probably some bad DNS server... Could you post the contents of /etc/resolv.conf when your laptop is connected to the Netgear? Here it is: cat /etc/resolv.conf # Generated by dhcpcd for interface wlan0 nameserver 192.168.1.1 Actually, 192.168.1.1 is the router, which is connected to the internet with a Netgear modem, IP 192.168.0.1 Alright. This means that (most likely) your laptop is asking your router to resolve rsync.gentoo.org and that router is giving back bad response(s), at least at first. You should check the configuration of the DNS server on the router. If you need further assistance, I'll need to know more about your router, specifically it's firm- and software. Also, if the router isn't running Gentoo, we may need to take this off-list of (at least) mark further messages as off-topic. If you want to confirm the problem is with the router, break out your generic DNS clients (dig/nslookup) and network monitoring tools (tcpdump et al.). Of course, your router my simply be giving bad responses because it's getting bad responses from further upstream. -- If there's one thing we've established over the years, it's that the vast majority of our users don't have the slightest clue what's best for them in terms of package stability. -- Gentoo Developer Ciaran McCreesh pgpMBOb9WNrbP.pgp Description: PGP signature
Re: [gentoo-user] emerge --sync connecting to 1.0.0.0 [OT, maybe]
On 12/16/06, Boyd Stephen Smith Jr. [EMAIL PROTECTED] wrote: On Saturday 16 December 2006 08:38, marco restelli [EMAIL PROTECTED] wrote about 'Re: [gentoo-user] emerge --sync connecting to 1.0.0.0 [OT, maybe]': On 12/16/06, Boyd Stephen Smith Jr. [EMAIL PROTECTED] wrote: Probably some bad DNS server... Could you post the contents of /etc/resolv.conf when your laptop is connected to the Netgear? Here it is: cat /etc/resolv.conf # Generated by dhcpcd for interface wlan0 nameserver 192.168.1.1 Actually, 192.168.1.1 is the router, which is connected to the internet with a Netgear modem, IP 192.168.0.1 Alright. This means that (most likely) your laptop is asking your router to resolve rsync.gentoo.org and that router is giving back bad response(s), at least at first. You should check the configuration of the DNS server on the router. Ok, this is already a useful information If you need further assistance, I'll need to know more about your router, specifically it's firm- and software. Also, if the router isn't running Gentoo, we may need to take this off-list of (at least) mark further messages as off-topic. It is Netgear WGR614 v6, and it is not running Gentoo. If you want to confirm the problem is with the router, break out your generic DNS clients (dig/nslookup) and network monitoring tools (tcpdump et al.). Of course, your router my simply be giving bad responses because it's getting bad responses from further upstream. Thanks for all the advices. Marco -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] emerge --sync connecting to 1.0.0.0 [OT, maybe]
On 12/16/06, marco restelli [EMAIL PROTECTED] wrote: On 12/16/06, Boyd Stephen Smith Jr. [EMAIL PROTECTED] wrote: On Saturday 16 December 2006 08:38, marco restelli [EMAIL PROTECTED] wrote about 'Re: [gentoo-user] emerge --sync connecting to 1.0.0.0 [OT, maybe]': On 12/16/06, Boyd Stephen Smith Jr. [EMAIL PROTECTED] wrote: Probably some bad DNS server... Could you post the contents of /etc/resolv.conf when your laptop is connected to the Netgear? Here it is: cat /etc/resolv.conf # Generated by dhcpcd for interface wlan0 nameserver 192.168.1.1 Actually, 192.168.1.1 is the router, which is connected to the internet with a Netgear modem, IP 192.168.0.1 Alright. This means that (most likely) your laptop is asking your router to resolve rsync.gentoo.org and that router is giving back bad response(s), at least at first. You should check the configuration of the DNS server on the router. Ok, this is already a useful information If you need further assistance, I'll need to know more about your router, specifically it's firm- and software. Also, if the router isn't running Gentoo, we may need to take this off-list of (at least) mark further messages as off-topic. It is Netgear WGR614 v6, and it is not running Gentoo. If you want to confirm the problem is with the router, break out your generic DNS clients (dig/nslookup) and network monitoring tools (tcpdump et al.). Of course, your router my simply be giving bad responses because it's getting bad responses from further upstream. Thanks for all the advices. Marco -- gentoo-user@gentoo.org mailing list I've seen that exact problem with many simple routers like the one you have and various Linux distros, almost always some of the programs do work with dns server being the router itself and some don't. Namely: Web browsers and IM apps mostly work, rsync,svn git and some telnet and ftp apps mostly don't. In MS Windows almost always all work just fine. The solution for me was to find out in the router itself what are the dns servers that it uses and configure them in the system, either in resolv.confor in conf.d/net . Hope it helps.
Re: [gentoo-user] emerge --sync fails
Thanks for your help friends!! I solved the problem with your advices. On 6/22/06, sternklang gentoo [EMAIL PROTECTED] wrote: Were you trying to do this?: PORTAGE_RSYNC_EXTRA_OPTS=--timeout=300 -- sternklang [EMAIL PROTECTED] -- Rafael Alfaro. Omnilife Independent Distributor. People taking care of people. -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] emerge --sync fails
On 6/22/06, Rafael Alfaro [EMAIL PROTECTED] wrote: PORTAGE_RSYNC_EXTRA_OPTS=300This is the problem, remove that from your /etc/make.conf - that's not an valid option... What were you trying to accomplish with it?Caster
Re: [gentoo-user] emerge --sync fails
Were you trying to do this?:PORTAGE_RSYNC_EXTRA_OPTS=--timeout=300-- sternklang[EMAIL PROTECTED]
Re: [gentoo-user] emerge --sync
On Sunday 21 May 2006 00:56, Leandro Melo de Sales wrote: Due to firewall configuration I can't make a emerge --sync using rsync, is it possible to do it using ftp or http? Try emerge-webrsync. It will use http. -- Bo Andresen pgpdjrSyWHrwS.pgp Description: PGP signature
Re: [gentoo-user] emerge --sync
2006/5/21, Leandro Melo de Sales [EMAIL PROTECTED]: Hi folks, Due to firewall configuration I can't make a emerge --sync usingrsync, is it possible to do it using ftp or http?emerge-webrsync
Re: [gentoo-user] emerge --sync
Leandro, emerge-webrsync should work. You can download a recent snapshot from one of the mirrors as an alternate. Patrick Hi folks, Due to firewall configuration I can't make a emerge --sync using rsync, is it possible to do it using ftp or http? Thanks, Leandro -- gentoo-user@gentoo.org mailing list -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Emerge sync says it's failing, but succeeds anyway. What's up with that?
Kevin O'Gorman wrote: Every time I emerge sync I get pretty much the same first few lines: rsync: connection unexpectedly closed (0 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(189) Welcome to cockatoo.gentoo.org http://cockatoo.gentoo.org Server Address : 65.19.163.230 http://65.19.163.230 Contact Name : [EMAIL PROTECTED] http://us.f805.mail.yahoo.com/ym/[EMAIL PROTECTED]YY=14693order=upsort=datepos=0view=ahead=b Hardware : 2 x Intel(R) Pentium(R) 4 CPU 2.80GHz, 1024MB RAM ... [and it goes on at some length to sync up just fine] Everything works well, but the top two lines make me wonder if I'm doing something wasteful. Clues, anyone? I just ran into this the other day. The rsync mirrors I was connecting to had an entry that didn't exist anymore. So, I filed this bug.. http://bugs.gentoo.org/show_bug.cgi?id=130283 Take a look at it and I think you can figure out what to do. If you can tell which server is failing then submit a bug and let them know. HTH, Jeremy -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] Emerge sync says it's failing, but succeeds anyway. What's up with that?
Thanks for filing the bug. It now is counted as resolved, and it may well be so, because I just tried emerge sync again, and the problem is gone. Thanks. BTW, does anyone know how this magic is done? Where are the site configurations kept -- is it all in the nameservers? ++ kevinOn 4/22/06, Jeremy Olexa [EMAIL PROTECTED] wrote: Kevin O'Gorman wrote: Every time I emerge sync I get pretty much the same first few lines: rsync: connection unexpectedly closed (0 bytes read so far) rsync error: error in rsync protocol data stream (code 12) at io.c(189) Welcome to cockatoo.gentoo.orghttp://cockatoo.gentoo.org Server Address : 65.19.163.230 http://65.19.163.230 Contact Name : [EMAIL PROTECTED] http://us.f805.mail.yahoo.com/ym/[EMAIL PROTECTED]YY=14693order=upsort=datepos=0view=ahead=b Hardware : 2 x Intel(R) Pentium(R) 4 CPU 2.80GHz, 1024MB RAM ... [and it goes on at some length to sync up just fine] Everything works well, but the top two lines make me wonder if I'm doing something wasteful. Clues, anyone? I just ran into this the other day. The rsync mirrors I was connectingto had an entry that didn't exist anymore. So, I filed this bug..http://bugs.gentoo.org/show_bug.cgi?id=130283 Take a look at it and I think you can figure out what to do. If you cantell which server is failing then submit a bug and let them know.HTH,Jeremy-- gentoo-user@gentoo.org mailing list-- Kevin O'Gorman, PhD
Re: [gentoo-user] emerge --sync error
Now I find the problem is caused by utf-8. My local changed from zh_CN.GB2312 to zh_CN.utf8 recently. But I don't know how to correct. -- Wang ShaoChun(王绍春) [EMAIL PROTECTED] PH.D Candidate Laboratory of Computer Science, Institute of Software Chinese Academy of Science -- gentoo-user@gentoo.org mailing list
