HEADS-UP: faster mirrors synching coming your way soon
I've just committed changes to pkg_create that will help mirrors synch by using much less bandwidth. I just ran a final test. Rsynching a full amd64 snapshot now says something like: sent 7,315,796,510 bytes received 40,292,721 bytes 4,517,095.01 bytes/sec total size is 28,752,806,019 speedup is 3.91 A few months ago, after the reorder files in packages, Stuart Henderson commented that this would not help mirrors, but just the end user, which got me thinking... (Reminder: archives are compressed files. rsync does not peek inside the compressed data, so its comparison algorithms don't work so well with them, as the first different byte will change everything for the rest of the archive, so no speed-up for compressed files). I looked at the --rsyncable patch for zlib/gzip, and talked it over with sthen@ and millert@, but pretty soon we discarded that idea. That patch is brittle (every zlib version has got its own flavor of it, with wild differences) and a nightmare to maintain. Plus it won't work at all with other compression formats. The solution was low-tech: simply cut the archive into more gzip chunks (signatures already split the package into two parts, so we know the tools work). I chose 16 files as a simple guideline to experiment with. There were still some discrepancies, such as tar timestamps metadata, which is why those migrated to the plist a few weeks ago (side-effect: the tarball effectively says everything dates back to the epoch... not so bad). I was pleasantly surprised: the size increase is minimal (very much under 1%). I also wacked on gzip timestamps, which don't serve any useful purpose either, especially since the plist signature also contains a timestamp (and that one is signed, so it's ways more trustworthy). Obviously, the first snapshot out will still copy everything. But from the second one, mirror owners should see a difference. To benefit: - mirror owners must now use rsync algorithms. Turn off -W / --whole-file if you were using it. - turn on -y / --fuzzy, as this will track minor package version changes. Note that this only applies to the package snapshots part of OpenBSD. My test was a bit extreme: I did build two snaps with the exact same ports tree, so the similarities are maximal. Nevertheless, there are lots of *huge* packages in the ports tree. So I expect the bandwidth gain to be very significant anyway, especially for fast architectures which turn up one snapshot a week or more. e.g., bandwidth use should be more than halved, I expect.
Re: Patch: fix high capacity ( 2GB) eMMC support
On Sun, September 21, 2014 2:09 pm, Raphael Graf wrote:
On Sat, September 20, 2014 7:45 pm, Jonathan Gray wrote:
On Sat, Sep 20, 2014 at 06:01:51PM +0200, Cédric Tessier wrote:
Hi,
I've bought a BeagleBone Black rev. C board, and I was trying to install
OpenBSD on it, but the internal eMMC was causing errors.
sdmmc1: unknown CARD_TYPE 0x17
scsibus1 at sdmmc1: 2 targets, initiator 0
sd1 at scsibus1 targ 1 lun 0: SD/MMC, Drive #01, SCSI2 0/direct fixed
sd1: 1024MB, 512 bytes/sector, 2097152 sectors
Card type and sectors count were wrong, and accessing the device was
causing I/O errors.
I've investigated the problem, and it looks like the support of
High Capacity eMMC ( 2GB) is missing.
I've written a quick and dirty patch (tested on 5.5 and snapshot) which fix
all my issues.
Modifications:
- mask reserved bits for card type value
These bits do not appear to be reserved. In your case bit 4
seems to indicate HS200/200 MHz clock capable.
Bit 5 is HS400/400 MHz clock capable.
- read sectors count from EXT_CSD
- fix sectors count and enable SDHC if High Capacity eMMC is detected
Is the old method of reading the block length still supported
with emmc and csd ver 2? That wasn't the case for normal sd card.
It seems the emmc situation is a bit different as the capacity
stored in a seperate place.
+
+ if (ext_csd[EXT_CSD_REV] = 2) {
+ sectors = ext_csd[EXT_CSD_SEC_COUNT + 0] 0 |
+ ext_csd[EXT_CSD_SEC_COUNT + 1]
8 |
+ ext_csd[EXT_CSD_SEC_COUNT + 2]
16 |
+ ext_csd[EXT_CSD_SEC_COUNT + 3]
24;
+ /*
+* High capacity MMC seems to report a magic 4096 *
512 bytes
+* capacity in csd, but ext_csd contains the real
sectors count
+*/
+ if ((sf-csd.capacity == (4096 * 512))
+ (sectors (2u * 1024 * 1024 * 1024) / 512)) {
+ sf-flags |= SFF_SDHC;
+ sf-csd.capacity = sectors;
+ }
I think this should change to
if (sectors (2u * 1024 * 1024 * 1024) / 512)
sf-flags |= SFF_SDHC;
sf-csd.capacity = sectors;
All csd rev 2 cards should report valid sectors in the extended space.
This seems not to be the case, the EXT_CSD_SEC_COUNT is only valid for
capacities over 2G. I'd use this field only if it's not zero, like freebsd
does.
Instead of masking the EXT_CSD_CARD_TYPE field, it may be better to just check
for the relevant bits as in the diff below.
Does this still work for you, Cédric?
Index: sys/dev/sdmmc/sdmmc_mem.c
===
RCS file: /cvs/src/sys/dev/sdmmc/sdmmc_mem.c,v
retrieving revision 1.19
diff -u -p -r1.19 sdmmc_mem.c
--- sys/dev/sdmmc/sdmmc_mem.c 12 Jul 2014 18:48:52 - 1.19
+++ sys/dev/sdmmc/sdmmc_mem.c 21 Sep 2014 12:02:35 -
@@ -428,6 +428,7 @@ sdmmc_mem_mmc_init(struct sdmmc_softc *s
u_int8_t ext_csd[512];
int speed = 0;
int hs_timing = 0;
+ u_int32_t sectors = 0;
if (sf-csd.mmcver = MMC_CSD_MMCVER_4_0) {
/* read EXT_CSD */
@@ -439,18 +440,12 @@ sdmmc_mem_mmc_init(struct sdmmc_softc *s
return error;
}
- switch (ext_csd[EXT_CSD_CARD_TYPE]) {
- case EXT_CSD_CARD_TYPE_26M:
- speed = 26000;
- break;
- case EXT_CSD_CARD_TYPE_52M:
- case EXT_CSD_CARD_TYPE_52M_V18:
- case EXT_CSD_CARD_TYPE_52M_V12:
- case EXT_CSD_CARD_TYPE_52M_V12_18:
+ if (ext_csd[EXT_CSD_CARD_TYPE] EXT_CSD_CARD_TYPE_52M) {
speed = 52000;
hs_timing = 1;
- break;
- default:
+ } else if (ext_csd[EXT_CSD_CARD_TYPE] EXT_CSD_CARD_TYPE_26M) {
+ speed = 26000;
+ } else {
printf(%s: unknown CARD_TYPE 0x%x\n, DEVNAME(sc),
ext_csd[EXT_CSD_CARD_TYPE]);
}
@@ -487,6 +482,16 @@ sdmmc_mem_mmc_init(struct sdmmc_softc *s
printf(%s, HS_TIMING set failed\n,
DEVNAME(sc));
return EINVAL;
}
+ }
+
+ sectors = ext_csd[EXT_CSD_SEC_COUNT + 0] 0 |
+ ext_csd[EXT_CSD_SEC_COUNT + 1] 8 |
+ ext_csd[EXT_CSD_SEC_COUNT + 2] 16 |
+ ext_csd[EXT_CSD_SEC_COUNT + 3] 24;
+
+ if (sectors != 0) {
+ sf-flags |= SFF_SDHC;
+ sf-csd.capacity = sectors;
}
}
Index:
Re: Patch: fix high capacity ( 2GB) eMMC support
On Mon, Sep 22, 2014 at 06:48:49PM +0200, Raphael Graf wrote:
I'm sorry, this diff was using the wrong defines.
Here's a corrected version.
Any comments or oks?
+
+ if (sectors != 0) {
+ sf-flags |= SFF_SDHC;
+ sf-csd.capacity = sectors;
}
I think I'd prefer to see the 2G test back here that the original diff
had. As the emmc spec says that sector addressing is only for 2G.
The lowest common nominator, 2GB in this case, will set the limit. The
implementation of a higher than 2GB of density of memory will not be
backwards compatible with the lower densities. First of all the address
argument for higher than 2GB of density of memory is changed to be
sector address (512B sectors) instead of byte address. Secondly the
density of the card is read from the EXT_CSD register instead of CSD
register.
And though I don't have any emmc using devices here the diff
otherwise looks good, so OK.
[patch] puc(4) add Winchiphead CH382 support
Hello,
Here is the patch to support Winchiphead CH382 PCIe-UART device.
I found the board at eBay with cheap price tag.
CH382 has three configurations and different PCI device ID.
- 2 serial (2S)
- 2 serial and 1 parallel (2S1P)
- 1 parallel (1P)
I have 2S board the patch does not support parallel port,
2S1P board will work as 2S.
After patching, CH382 is recognized as 16750 like this.
puc0 at pci3 dev 0 function 0 Nanjing QinHeng Electronics CH382 rev 0x10: port
s: 2 com
com4 at puc0 port 0 apic 5 int 17: ti16750, 64 byte fifo
com5 at puc0 port 1 apic 5 int 17: ti16750, 64 byte fifo
I referred the following pages:
http://www.spinics.net/lists/linux-serial/msg11744.html
http://kent-vandervelden.blogspot.jp/2014/08/linux-parallel-port-cards.html
Can I commit?
Regards,
--
SASANO Takayoshi u...@mx5.nisiq.net
Index: pucdata.c
===
RCS file: /cvs/src/sys/dev/pci/pucdata.c,v
retrieving revision 1.93
diff -u -p -r1.93 pucdata.c
--- pucdata.c 13 Aug 2014 07:45:37 - 1.93
+++ pucdata.c 21 Sep 2014 12:28:26 -
@@ -2080,6 +2080,22 @@ const struct puc_device_description puc_
{ PUC_COM_POW2(0), 0x14, 0x },
},
},
+ { /* WinChipHead CH382 (2S), */
+ { PCI_VENDOR_WCH2, PCI_PRODUCT_WCH2_CH382_1, 0, 0},
+ { 0x, 0x, 0, 0},
+ {
+ { PUC_COM_POW2(0), 0x10, 0x00c0 },
+ { PUC_COM_POW2(0), 0x10, 0x00c8 },
+ },
+ },
+ { /* WinChipHead CH382 (2S1P), */
+ { PCI_VENDOR_WCH2, PCI_PRODUCT_WCH2_CH382_2, 0, 0},
+ { 0x, 0x, 0, 0},
+ {
+ { PUC_COM_POW2(0), 0x10, 0x00c0 },
+ { PUC_COM_POW2(0), 0x10, 0x00c8 },
+ },
+ },
{ /* NetMos NM9820 UART */
{ PCI_VENDOR_NETMOS, PCI_PRODUCT_NETMOS_NM9820, 0, 0},
{ 0x, 0x, 0, 0},
Re: [patch] puc(4) add Winchiphead CH382 support
Date: Tue, 23 Sep 2014 05:44:04 +0900
From: SASANO Takayoshi u...@mx5.nisiq.net
Hello,
Here is the patch to support Winchiphead CH382 PCIe-UART device.
I found the board at eBay with cheap price tag.
CH382 has three configurations and different PCI device ID.
- 2 serial (2S)
- 2 serial and 1 parallel (2S1P)
- 1 parallel (1P)
I have 2S board the patch does not support parallel port,
2S1P board will work as 2S.
After patching, CH382 is recognized as 16750 like this.
puc0 at pci3 dev 0 function 0 Nanjing QinHeng Electronics CH382 rev 0x10:
port
s: 2 com
com4 at puc0 port 0 apic 5 int 17: ti16750, 64 byte fifo
com5 at puc0 port 1 apic 5 int 17: ti16750, 64 byte fifo
I referred the following pages:
http://www.spinics.net/lists/linux-serial/msg11744.html
http://kent-vandervelden.blogspot.jp/2014/08/linux-parallel-port-cards.html
Can I commit?
ok kettenis@
Regards,
--
SASANO Takayoshi u...@mx5.nisiq.net
Index: pucdata.c
===
RCS file: /cvs/src/sys/dev/pci/pucdata.c,v
retrieving revision 1.93
diff -u -p -r1.93 pucdata.c
--- pucdata.c 13 Aug 2014 07:45:37 - 1.93
+++ pucdata.c 21 Sep 2014 12:28:26 -
@@ -2080,6 +2080,22 @@ const struct puc_device_description puc_
{ PUC_COM_POW2(0), 0x14, 0x },
},
},
+ { /* WinChipHead CH382 (2S), */
+ { PCI_VENDOR_WCH2, PCI_PRODUCT_WCH2_CH382_1, 0, 0},
+ { 0x, 0x, 0, 0},
+ {
+ { PUC_COM_POW2(0), 0x10, 0x00c0 },
+ { PUC_COM_POW2(0), 0x10, 0x00c8 },
+ },
+ },
+ { /* WinChipHead CH382 (2S1P), */
+ { PCI_VENDOR_WCH2, PCI_PRODUCT_WCH2_CH382_2, 0, 0},
+ { 0x, 0x, 0, 0},
+ {
+ { PUC_COM_POW2(0), 0x10, 0x00c0 },
+ { PUC_COM_POW2(0), 0x10, 0x00c8 },
+ },
+ },
{ /* NetMos NM9820 UART */
{ PCI_VENDOR_NETMOS, PCI_PRODUCT_NETMOS_NM9820, 0, 0},
{ 0x, 0x, 0, 0},
sm(4) is too verbose
sm(4) doenst have to bpfdetach cos if_detach will do it for it. also, useless comments are useless. ok? Index: dev/ic/smc91cxx.c === RCS file: /cvs/src/sys/dev/ic/smc91cxx.c,v retrieving revision 1.35 diff -u -p -r1.35 smc91cxx.c --- dev/ic/smc91cxx.c 22 Jul 2014 13:12:12 - 1.35 +++ dev/ic/smc91cxx.c 23 Sep 2014 01:20:29 - @@ -697,7 +697,6 @@ smc91cxx_start(ifp) ifp-if_timer = 5; #if NBPFILTER 0 - /* Hand off a copy to the bpf. */ if (ifp-if_bpf) bpf_mtap(ifp-if_bpf, top, BPF_DIRECTION_OUT); #endif @@ -997,10 +996,6 @@ smc91cxx_read(sc) ifp-if_ipackets++; #if NBPFILTER 0 - /* -* Hand the packet off to bpf listeners. If there's a bpf listener, -* we need to check if the packet is ours. -*/ if (ifp-if_bpf) bpf_mtap(ifp-if_bpf, m, BPF_DIRECTION_IN); #endif @@ -1239,9 +1234,6 @@ smc91cxx_detach(self, flags) /* Delete all media. */ ifmedia_delete_instance(sc-sc_mii.mii_media, IFM_INST_ANY); -#if NBPFILTER 0 - bpfdetach(ifp); -#endif ether_ifdetach(ifp); if_detach(ifp);
