Re: Using MCLGETI for sk(4)
On Fri, May 18, 2012 at 5:16 AM, Brad Smith b...@comstyle.com wrote: On Mon, May 07, 2012 at 10:24:37AM +0100, Stuart Henderson wrote: On 2012/05/06 21:34, Brad Smith wrote: I have resurrected the old jumbo allocator to MCLGETI conversion diff that was reverted with rev 1.152. The commit did not indicate why that was so. I updated it to -current and have tested it a fair bit on amd64 with a SysKonnect GEnesis board using jumbos and have not found any issues so far. Theo or Stuart can you comment on why this was reverted? Forwarded you some mails offlist, TL;DR version is it dies under heavy packet load. Would be nice to have this back as the jumbo allocator uses a huge wodge of kvm. Not sure if I have a good machine for testing it any more. Ya, so I've heard but cannot reproduce any issues so far. Also test with a Yukon board with multiple ping -f's in both directions. Up to 53 jumbo mbuf clusters allocated so far. skc0 at pci1 dev 7 function 0 Schneider Koch SK-98xx v2.0 rev 0x20, Yukon (0x1): apic 2 int 16 sk0 at skc0 port A: address 00:00:5a:9f:31:32 eephy0 at sk0 phy 0: 88E1011 Gigabit PHY, rev. 3 It appears that MCLGETI() is affected by different chipset revisions . Testing all the revisions is difficult to do. That's why my MCLGETI diffs still sleep in my local tree :-) -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean. -- Brightest day, Blackest night, No bug shall escape my sight, And those who worship evil's mind, be wary of my powers, puffy lantern's light !
Re: Using MCLGETI for sk(4)
On 2012/05/19 19:09, Loganaden Velvindron wrote: On Fri, May 18, 2012 at 5:16 AM, Brad Smith b...@comstyle.com wrote: On Mon, May 07, 2012 at 10:24:37AM +0100, Stuart Henderson wrote: On 2012/05/06 21:34, Brad Smith wrote: I have resurrected the old jumbo allocator to MCLGETI conversion diff that was reverted with rev 1.152. The commit did not indicate why that was so. I updated it to -current and have tested it a fair bit on amd64 with a SysKonnect GEnesis board using jumbos and have not found any issues so far. Theo or Stuart can you comment on why this was reverted? Forwarded you some mails offlist, TL;DR version is it dies under heavy packet load. Would be nice to have this back as the jumbo allocator uses a huge wodge of kvm. Not sure if I have a good machine for testing it any more. Ya, so I've heard but cannot reproduce any issues so far. Also test with a Yukon board with multiple ping -f's in both directions. Up to 53 jumbo mbuf clusters allocated so far. skc0 at pci1 dev 7 function 0 Schneider Koch SK-98xx v2.0 rev 0x20, Yukon (0x1): apic 2 int 16 sk0 at skc0 port A: address 00:00:5a:9f:31:32 eephy0 at sk0 phy 0: 88E1011 Gigabit PHY, rev. 3 It appears that MCLGETI() is affected by different chipset revisions . Testing all the revisions is difficult to do. That's why my MCLGETI diffs still sleep in my local tree :-) Looking at old dmesg, mine would have been this one, skc0 at pci1 dev 6 function 0 3Com 3c940 rev 0x10, Yukon (0x1): apic 2 int 3 (irq 3) sk0 at skc0 port A: address 00:0a:5e:1a:a3:00 eephy0 at sk0 phy 0: Marvell 88E1011 Gigabit PHY, rev. 3 Not sure what card gollo@ was using.
Re: Using MCLGETI for sk(4)
On 2012/05/17 21:16, Brad Smith wrote: On Mon, May 07, 2012 at 10:24:37AM +0100, Stuart Henderson wrote: On 2012/05/06 21:34, Brad Smith wrote: I have resurrected the old jumbo allocator to MCLGETI conversion diff that was reverted with rev 1.152. The commit did not indicate why that was so. I updated it to -current and have tested it a fair bit on amd64 with a SysKonnect GEnesis board using jumbos and have not found any issues so far. Theo or Stuart can you comment on why this was reverted? Forwarded you some mails offlist, TL;DR version is it dies under heavy packet load. Would be nice to have this back as the jumbo allocator uses a huge wodge of kvm. Not sure if I have a good machine for testing it any more. Ya, so I've heard but cannot reproduce any issues so far. Also test with a Yukon board with multiple ping -f's in both directions. Up to 53 jumbo mbuf clusters allocated so far. skc0 at pci1 dev 7 function 0 Schneider Koch SK-98xx v2.0 rev 0x20, Yukon (0x1): apic 2 int 16 sk0 at skc0 port A: address 00:00:5a:9f:31:32 eephy0 at sk0 phy 0: 88E1011 Gigabit PHY, rev. 3 ping -f is relatively polite on the network, try something nastier like pkg_add netrate; netblast $ip_addr $port 1 60 or tcpbench -B 1 -u $host.
Re: Using MCLGETI for sk(4)
On Mon, May 07, 2012 at 10:24:37AM +0100, Stuart Henderson wrote: On 2012/05/06 21:34, Brad Smith wrote: I have resurrected the old jumbo allocator to MCLGETI conversion diff that was reverted with rev 1.152. The commit did not indicate why that was so. I updated it to -current and have tested it a fair bit on amd64 with a SysKonnect GEnesis board using jumbos and have not found any issues so far. Theo or Stuart can you comment on why this was reverted? Forwarded you some mails offlist, TL;DR version is it dies under heavy packet load. Would be nice to have this back as the jumbo allocator uses a huge wodge of kvm. Not sure if I have a good machine for testing it any more. Ya, so I've heard but cannot reproduce any issues so far. Also test with a Yukon board with multiple ping -f's in both directions. Up to 53 jumbo mbuf clusters allocated so far. skc0 at pci1 dev 7 function 0 Schneider Koch SK-98xx v2.0 rev 0x20, Yukon (0x1): apic 2 int 16 sk0 at skc0 port A: address 00:00:5a:9f:31:32 eephy0 at sk0 phy 0: 88E1011 Gigabit PHY, rev. 3 -- This message has been scanned for viruses and dangerous content by MailScanner, and is believed to be clean.
Re: Using MCLGETI for sk(4)
On 2012/05/06 21:34, Brad Smith wrote: I have resurrected the old jumbo allocator to MCLGETI conversion diff that was reverted with rev 1.152. The commit did not indicate why that was so. I updated it to -current and have tested it a fair bit on amd64 with a SysKonnect GEnesis board using jumbos and have not found any issues so far. Theo or Stuart can you comment on why this was reverted? Forwarded you some mails offlist, TL;DR version is it dies under heavy packet load. Would be nice to have this back as the jumbo allocator uses a huge wodge of kvm. Not sure if I have a good machine for testing it any more.
Using MCLGETI for sk(4)
I have resurrected the old jumbo allocator to MCLGETI conversion diff
that was reverted with rev 1.152. The commit did not indicate why that
was so. I updated it to -current and have tested it a fair bit on amd64
with a SysKonnect GEnesis board using jumbos and have not found any issues
so far. Theo or Stuart can you comment on why this was reverted?
This affects users whether using jumbos or not. Can anyone else with sk(4)
please test this?
skc0 at pci1 dev 7 function 0 Schneider Koch SK-98xx rev 0x13, GEnesis
(0x0): apic 2 int 16
sk0 at skc0 port A: address 00:00:5a:98:b9:c0
brgphy0 at sk0 phy 1: BCM5400 1000baseT PHY, rev. 7
Index: if_sk.c
===
RCS file: /cvs/src/sys/dev/pci/if_sk.c,v
retrieving revision 1.161
diff -u -p -r1.161 if_sk.c
--- if_sk.c 24 Feb 2012 06:19:00 - 1.161
+++ if_sk.c 4 May 2012 03:16:30 -
@@ -156,12 +156,10 @@ void sk_watchdog(struct ifnet *);
int sk_ifmedia_upd(struct ifnet *);
void sk_ifmedia_sts(struct ifnet *, struct ifmediareq *);
void sk_reset(struct sk_softc *);
-int sk_newbuf(struct sk_if_softc *, int, struct mbuf *, bus_dmamap_t);
-int sk_alloc_jumbo_mem(struct sk_if_softc *);
-void *sk_jalloc(struct sk_if_softc *);
-void sk_jfree(caddr_t, u_int, void *);
+int sk_newbuf(struct sk_if_softc *);
int sk_init_rx_ring(struct sk_if_softc *);
int sk_init_tx_ring(struct sk_if_softc *);
+void sk_fill_rx_ring(struct sk_if_softc *);
int sk_xmac_miibus_readreg(struct device *, int, int);
void sk_xmac_miibus_writereg(struct device *, int, int, int);
@@ -573,21 +571,23 @@ sk_init_rx_ring(struct sk_if_softc *sc_i
sizeof(struct ip));
}
- for (i = 0; i SK_RX_RING_CNT; i++) {
- if (sk_newbuf(sc_if, i, NULL,
- sc_if-sk_cdata.sk_rx_jumbo_map) == ENOBUFS) {
- printf(%s: failed alloc of %dth mbuf\n,
- sc_if-sk_dev.dv_xname, i);
- return (ENOBUFS);
- }
- }
-
sc_if-sk_cdata.sk_rx_prod = 0;
sc_if-sk_cdata.sk_rx_cons = 0;
+ sc_if-sk_cdata.sk_rx_cnt = 0;
+ sk_fill_rx_ring(sc_if);
return (0);
}
+void
+sk_fill_rx_ring(struct sk_if_softc *sc_if)
+{
+ while (sc_if-sk_cdata.sk_rx_cnt SK_RX_RING_CNT) {
+ if (sk_newbuf(sc_if) == ENOBUFS)
+ break;
+ }
+}
+
int
sk_init_tx_ring(struct sk_if_softc *sc_if)
{
@@ -635,199 +635,60 @@ sk_init_tx_ring(struct sk_if_softc *sc_i
}
int
-sk_newbuf(struct sk_if_softc *sc_if, int i, struct mbuf *m,
- bus_dmamap_t dmamap)
+sk_newbuf(struct sk_if_softc *sc_if)
{
- struct mbuf *m_new = NULL;
struct sk_chain *c;
struct sk_rx_desc *r;
+ struct mbuf *m;
+ bus_dmamap_tdmamap;
+ u_int32_t sk_ctl;
+ int i, error;
- if (m == NULL) {
- caddr_t buf = NULL;
+ m = MCLGETI(NULL, M_DONTWAIT, sc_if-arpcom.ac_if, SK_JLEN);
+ if (!m)
+ return (ENOBUFS);
+ m-m_len = m-m_pkthdr.len = SK_JLEN;
+ m_adj(m, ETHER_ALIGN);
- MGETHDR(m_new, M_DONTWAIT, MT_DATA);
- if (m_new == NULL)
- return (ENOBUFS);
+ dmamap = sc_if-sk_cdata.sk_rx_map[sc_if-sk_cdata.sk_rx_prod];
- /* Allocate the jumbo buffer */
- buf = sk_jalloc(sc_if);
- if (buf == NULL) {
- m_freem(m_new);
- DPRINTFN(1, (%s jumbo allocation failed -- packet
- dropped!\n, sc_if-arpcom.ac_if.if_xname));
- return (ENOBUFS);
- }
-
- /* Attach the buffer to the mbuf */
- m_new-m_len = m_new-m_pkthdr.len = SK_JLEN;
- MEXTADD(m_new, buf, SK_JLEN, 0, sk_jfree, sc_if);
- } else {
- /*
-* We're re-using a previously allocated mbuf;
-* be sure to re-init pointers and lengths to
-* default values.
-*/
- m_new = m;
- m_new-m_len = m_new-m_pkthdr.len = SK_JLEN;
- m_new-m_data = m_new-m_ext.ext_buf;
+ error = bus_dmamap_load_mbuf(sc_if-sk_softc-sc_dmatag, dmamap, m,
+ BUS_DMA_READ|BUS_DMA_NOWAIT);
+ if (error) {
+ m_freem(m);
+ return (ENOBUFS);
}
- m_adj(m_new, ETHER_ALIGN);
- c = sc_if-sk_cdata.sk_rx_chain[i];
- r = c-sk_desc;
- c-sk_mbuf = m_new;
- r-sk_data_lo = htole32(dmamap-dm_segs[0].ds_addr +
- (((vaddr_t)m_new-m_data
- - (vaddr_t)sc_if-sk_cdata.sk_jumbo_buf)));
- r-sk_ctl = htole32(SK_JLEN | SK_RXSTAT);
-
- SK_CDRXSYNC(sc_if, i, BUS_DMASYNC_PREWRITE|BUS_DMASYNC_PREREAD);
-
- return (0);
-}
-
-/*
- * Memory
