Is the setup and teardown per transfer or when file is opened and closed? Or is it set up once per context switch of task?
I am partly interested cos I would like to improve mt one day (as user of tape and Sparc64 Txxx) if I get the time. Andrew On Fri, 19 Oct 2018 at 10:22, Mark Kettenis <[email protected]> wrote: > > Date: Fri, 19 Oct 2018 10:22:30 +1000 > > From: David Gwynne <[email protected]> > > > > On Wed, May 10, 2017 at 10:09:59PM +1000, David Gwynne wrote: > > > On Mon, May 08, 2017 at 11:03:58AM +1000, David Gwynne wrote: > > > > on modern sparc64s (think fire or sparc enterprise Mx000 boxes), > > > > setting up and tearing down the translation table entries (TTEs) > > > > is very expensive. so expensive that the cost of doing it for disk > > > > io has a noticable impact on compile times. > > > > > > > > now that there's a BUS_DMA_64BIT flag, we can use that to decide > > > > to bypass the iommu for devices that set that flag, therefore > > > > avoiding the cost of handling the TTEs. > > > > > > > > the following diff adds support for bypass mappings to the iommu > > > > code on sparc64. it's based on a diff from kettenis@ back in 2009. > > > > the main changes are around coping with the differences between > > > > schizo/psycho and fire/oberon. > > > > > > > > the differences between the chips are now represented by a iommu_hw > > > > struct. these differences include how to enable the iommu (now via > > > > a function pointer), and masks for bypass addresses. > > > > > > > > ive tested this on oberon (on an m4000) and schizo (on a v880). > > > > however, the bypass code isnt working on fire (v245s). to cope with > > > > that for now, the iommu_hw struct lets drivers mask flag bits that > > > > are handled when creating a dmamap. this means fire boards will > > > > ignore BUS_DMA_64BIT until i can figure out whats wrong with them. > > > > > > i figured it out. it turns out Fire was working fine. however, > > > enabling 64bit dva on the onboard devices didnt work because the > > > serverworks/broadcom pcie to pcix bridge can only handle dma addresses > > > in the low 40 bits. because the fire bypass window is higher than > > > this, the bridge would choke and things stopped working. > > > > > > the updated diff attempts to handle this. basically when probing > > > the bridge, the platform creates a custom dma tag for it. this tag > > > intercets bus_dmamap_create and clears the BUS_DMA_64BIT flag before > > > handing it up to the parent bridge, which is pyro in my situation. > > > it looks like early sun4v boxes could make use of this too. > > > > > > > i have not tested this on psycho yet. if anyone has such a machine > > > > and is willing to work with me to figure it out, please talk to me. > > > > > > i still dont have psycho reports. > > > > Would anyone object if I committed this? I've been running it for the > > last release or two without issues, but with significant improvements in > > performance on the machines involved. > > At the price of giving all PCI devices unrestricted access to memory. > > So I'm not eager to this, especially since on sun4v hardware bypassing > the iommu isn't possible as soon as multiple domains are enabled. And > we lose a useful diagnostic when developing drivers. Are you sure the > iommu overhead can't be reduced some other way? At some point we > probably want to add iommu support on amd64 and arm64, but if that > comes with a similar overhead as on sparc64 that's going to be a bit > of an issue. > > > > Index: dev/iommu.c > > > =================================================================== > > > RCS file: /cvs/src/sys/arch/sparc64/dev/iommu.c,v > > > retrieving revision 1.74 > > > diff -u -p -r1.74 iommu.c > > > --- dev/iommu.c 30 Apr 2017 16:45:45 -0000 1.74 > > > +++ dev/iommu.c 10 May 2017 12:00:09 -0000 > > > @@ -100,6 +100,25 @@ void iommu_iomap_clear_pages(struct iomm > > > void _iommu_dvmamap_sync(bus_dma_tag_t, bus_dma_tag_t, bus_dmamap_t, > > > bus_addr_t, bus_size_t, int); > > > > > > +void iommu_hw_enable(struct iommu_state *); > > > + > > > +const struct iommu_hw iommu_hw_default = { > > > + .ihw_enable = iommu_hw_enable, > > > + > > > + .ihw_dvma_pa = IOTTE_PAMASK, > > > + > > > + .ihw_bypass = 0x3fffUL << 50, > > > + .ihw_bypass_nc = 0, > > > + .ihw_bypass_ro = 0, > > > +}; > > > + > > > +void > > > +iommu_hw_enable(struct iommu_state *is) > > > +{ > > > + IOMMUREG_WRITE(is, iommu_tsb, is->is_ptsb); > > > + IOMMUREG_WRITE(is, iommu_cr, IOMMUCR_EN | (is->is_tsbsize << 16)); > > > +} > > > + > > > /* > > > * Initiate an STC entry flush. > > > */ > > > @@ -125,7 +144,8 @@ iommu_strbuf_flush(struct strbuf_ctl *sb > > > * - create a private DVMA map. > > > */ > > > void > > > -iommu_init(char *name, struct iommu_state *is, int tsbsize, u_int32_t > iovabase) > > > +iommu_init(char *name, const struct iommu_hw *ihw, struct iommu_state > *is, > > > + int tsbsize, u_int32_t iovabase) > > > { > > > psize_t size; > > > vaddr_t va; > > > @@ -149,13 +169,9 @@ iommu_init(char *name, struct iommu_stat > > > * be hard-wired, so we read the start and size from the PROM and > > > * just use those values. > > > */ > > > - if (strncmp(name, "pyro", 4) == 0) { > > > - is->is_cr = IOMMUREG_READ(is, iommu_cr); > > > - is->is_cr &= ~IOMMUCR_FIRE_BE; > > > - is->is_cr |= (IOMMUCR_FIRE_SE | IOMMUCR_FIRE_CM_EN | > > > - IOMMUCR_FIRE_TE); > > > - } else > > > - is->is_cr = IOMMUCR_EN; > > > + > > > + is->is_hw = ihw; > > > + > > > is->is_tsbsize = tsbsize; > > > if (iovabase == (u_int32_t)-1) { > > > is->is_dvmabase = IOTSB_VSTART(is->is_tsbsize); > > > @@ -237,15 +253,6 @@ iommu_init(char *name, struct iommu_stat > > > mtx_init(&is->is_mtx, IPL_HIGH); > > > > > > /* > > > - * Set the TSB size. The relevant bits were moved to the TSB > > > - * base register in the PCIe host bridges. > > > - */ > > > - if (strncmp(name, "pyro", 4) == 0) > > > - is->is_ptsb |= is->is_tsbsize; > > > - else > > > - is->is_cr |= (is->is_tsbsize << 16); > > > - > > > - /* > > > * Now actually start up the IOMMU. > > > */ > > > iommu_reset(is); > > > @@ -262,10 +269,7 @@ iommu_reset(struct iommu_state *is) > > > { > > > int i; > > > > > > - IOMMUREG_WRITE(is, iommu_tsb, is->is_ptsb); > > > - > > > - /* Enable IOMMU */ > > > - IOMMUREG_WRITE(is, iommu_cr, is->is_cr); > > > + (*is->is_hw->ihw_enable)(is); > > > > > > for (i = 0; i < 2; ++i) { > > > struct strbuf_ctl *sb = is->is_sb[i]; > > > @@ -280,7 +284,7 @@ iommu_reset(struct iommu_state *is) > > > printf(", STC%d enabled", i); > > > } > > > > > > - if (is->is_flags & IOMMU_FLUSH_CACHE) > > > + if (ISSET(is->is_hw->ihw_flags, IOMMU_HW_FLUSH_CACHE)) > > > IOMMUREG_WRITE(is, iommu_cache_invalidate, -1ULL); > > > } > > > > > > @@ -433,7 +437,7 @@ iommu_extract(struct iommu_state *is, bu > > > if (dva >= is->is_dvmabase && dva <= is->is_dvmaend) > > > tte = is->is_tsb[IOTSBSLOT(dva, is->is_tsbsize)]; > > > > > > - return (tte & IOTTE_PAMASK); > > > + return (tte & is->is_hw->ihw_dvma_pa); > > > } > > > > > > /* > > > @@ -601,6 +605,7 @@ iommu_dvmamap_create(bus_dma_tag_t t, bu > > > { > > > int ret; > > > bus_dmamap_t map; > > > + struct iommu_state *is = sb->sb_iommu; > > > struct iommu_map_state *ims; > > > > > > BUS_DMA_FIND_PARENT(t, _dmamap_create); > > > @@ -610,6 +615,12 @@ iommu_dvmamap_create(bus_dma_tag_t t, bu > > > if (ret) > > > return (ret); > > > > > > + if (flags & BUS_DMA_64BIT) { > > > + map->_dm_cookie = is; > > > + *dmamap = map; > > > + return (0); > > > + } > > > + > > > ims = iommu_iomap_create(atop(round_page(size))); > > > > > > if (ims == NULL) { > > > @@ -641,8 +652,10 @@ iommu_dvmamap_destroy(bus_dma_tag_t t, b > > > if (map->dm_nsegs) > > > bus_dmamap_unload(t0, map); > > > > > > - if (map->_dm_cookie) > > > - iommu_iomap_destroy(map->_dm_cookie); > > > + if (!ISSET(map->_dm_flags, BUS_DMA_64BIT)) { > > > + if (map->_dm_cookie) > > > + iommu_iomap_destroy(map->_dm_cookie); > > > + } > > > map->_dm_cookie = NULL; > > > > > > BUS_DMA_FIND_PARENT(t, _dmamap_destroy); > > > @@ -667,36 +680,36 @@ iommu_dvmamap_load(bus_dma_tag_t t, bus_ > > > u_long dvmaddr, sgstart, sgend; > > > bus_size_t align, boundary; > > > struct iommu_state *is; > > > - struct iommu_map_state *ims = map->_dm_cookie; > > > + struct iommu_map_state *ims; > > > pmap_t pmap; > > > > > > -#ifdef DIAGNOSTIC > > > - if (ims == NULL) > > > - panic("iommu_dvmamap_load: null map state"); > > > -#endif > > > -#ifdef DEBUG > > > - if (ims->ims_sb == NULL) > > > - panic("iommu_dvmamap_load: null sb"); > > > - if (ims->ims_sb->sb_iommu == NULL) > > > - panic("iommu_dvmamap_load: null iommu"); > > > -#endif /* DEBUG */ > > > - is = ims->ims_sb->sb_iommu; > > > - > > > - if (map->dm_nsegs) { > > > - /* > > > - * Is it still in use? _bus_dmamap_load should have taken > care > > > - * of this. > > > - */ > > > -#ifdef DIAGNOSTIC > > > - panic("iommu_dvmamap_load: map still in use"); > > > -#endif > > > - bus_dmamap_unload(t0, map); > > > - } > > > - > > > /* > > > * Make sure that on error condition we return "no valid mappings". > > > */ > > > - map->dm_nsegs = 0; > > > + KASSERTMSG(map->dm_nsegs == 0, "map still in use"); > > > + > > > + if (ISSET(map->_dm_flags, BUS_DMA_64BIT)) { > > > + unsigned long bypass; > > > + int i; > > > + > > > + is = map->_dm_cookie; > > > + bypass = is->is_hw->ihw_bypass; > > > + > > > + /* Bypass translation by the IOMMU. */ > > > + > > > + BUS_DMA_FIND_PARENT(t, _dmamap_load); > > > + err = (*t->_dmamap_load)(t, t0, map, buf, buflen, p, > flags); > > > + if (err != 0) > > > + return (err); > > > + > > > + for (i = 0; i < map->dm_nsegs; i++) > > > + map->dm_segs[i].ds_addr |= bypass; > > > + > > > + return (0); > > > + } > > > + > > > + ims = map->_dm_cookie; > > > + is = ims->ims_sb->sb_iommu; > > > > > > if (buflen < 1 || buflen > map->_dm_size) { > > > DPRINTF(IDB_BUSDMA, > > > @@ -876,28 +889,31 @@ iommu_dvmamap_load_raw(bus_dma_tag_t t, > > > bus_size_t boundary, align; > > > u_long dvmaddr, sgstart, sgend; > > > struct iommu_state *is; > > > - struct iommu_map_state *ims = map->_dm_cookie; > > > + struct iommu_map_state *ims; > > > > > > -#ifdef DIAGNOSTIC > > > - if (ims == NULL) > > > - panic("iommu_dvmamap_load_raw: null map state"); > > > -#endif > > > -#ifdef DEBUG > > > - if (ims->ims_sb == NULL) > > > - panic("iommu_dvmamap_load_raw: null sb"); > > > - if (ims->ims_sb->sb_iommu == NULL) > > > - panic("iommu_dvmamap_load_raw: null iommu"); > > > -#endif /* DEBUG */ > > > - is = ims->ims_sb->sb_iommu; > > > + KASSERTMSG(map->dm_nsegs == 0, "map stil in use"); > > > > > > - if (map->dm_nsegs) { > > > - /* Already in use?? */ > > > -#ifdef DIAGNOSTIC > > > - panic("iommu_dvmamap_load_raw: map still in use"); > > > -#endif > > > - bus_dmamap_unload(t0, map); > > > + if (ISSET(map->_dm_flags, BUS_DMA_64BIT)) { > > > + unsigned long bypass; > > > + > > > + is = map->_dm_cookie; > > > + bypass = is->is_hw->ihw_bypass; > > > + > > > + /* Bypass translation by the IOMMU. */ > > > + for (i = 0; i < nsegs; i++) { > > > + map->dm_segs[i].ds_addr = bypass | segs[i].ds_addr; > > > + map->dm_segs[i].ds_len = segs[i].ds_len; > > > + } > > > + > > > + map->dm_nsegs = nsegs; > > > + map->dm_mapsize = size; > > > + > > > + return (0); > > > } > > > > > > + ims = map->_dm_cookie; > > > + is = ims->ims_sb->sb_iommu; > > > + > > > /* > > > * A boundary presented to bus_dmamem_alloc() takes precedence > > > * over boundary in the map. > > > @@ -1088,11 +1104,6 @@ iommu_dvmamap_append_range(bus_dma_tag_t > > > bus_dma_segment_t *seg = NULL; > > > int i = map->dm_nsegs; > > > > > > -#ifdef DEBUG > > > - if (ims == NULL) > > > - panic("iommu_dvmamap_append_range: null map state"); > > > -#endif > > > - > > > sgstart = iommu_iomap_translate(ims, pa); > > > sgend = sgstart + length - 1; > > > > > > @@ -1298,20 +1309,17 @@ void > > > iommu_dvmamap_unload(bus_dma_tag_t t, bus_dma_tag_t t0, bus_dmamap_t > map) > > > { > > > struct iommu_state *is; > > > - struct iommu_map_state *ims = map->_dm_cookie; > > > + struct iommu_map_state *ims; > > > bus_addr_t dvmaddr = map->_dm_dvmastart; > > > bus_size_t sgsize = map->_dm_dvmasize; > > > int error; > > > > > > -#ifdef DEBUG > > > - if (ims == NULL) > > > - panic("iommu_dvmamap_unload: null map state"); > > > - if (ims->ims_sb == NULL) > > > - panic("iommu_dvmamap_unload: null sb"); > > > - if (ims->ims_sb->sb_iommu == NULL) > > > - panic("iommu_dvmamap_unload: null iommu"); > > > -#endif /* DEBUG */ > > > + if (ISSET(map->_dm_flags, BUS_DMA_64BIT)) { > > > + bus_dmamap_unload(t->_parent, map); > > > + return; > > > + } > > > > > > + ims = map->_dm_cookie; > > > is = ims->ims_sb->sb_iommu; > > > > > > /* Flush the iommu */ > > > @@ -1488,7 +1496,7 @@ iommu_dvmamap_print_map(bus_dma_tag_t t, > > > break; > > > } > > > > > > - if (map->_dm_cookie) { > > > + if (!ISSET(map->_dm_flags, BUS_DMA_64BIT) && map->_dm_cookie != > NULL) { > > > struct iommu_map_state *ims = map->_dm_cookie; > > > struct iommu_page_map *ipm = &ims->ims_map; > > > > > > @@ -1546,19 +1554,19 @@ void > > > iommu_dvmamap_sync(bus_dma_tag_t t, bus_dma_tag_t t0, bus_dmamap_t > map, > > > bus_addr_t offset, bus_size_t len, int ops) > > > { > > > - struct iommu_map_state *ims = map->_dm_cookie; > > > + struct iommu_map_state *ims; > > > > > > -#ifdef DIAGNOSTIC > > > - if (ims == NULL) > > > - panic("iommu_dvmamap_sync: null map state"); > > > - if (ims->ims_sb == NULL) > > > - panic("iommu_dvmamap_sync: null sb"); > > > - if (ims->ims_sb->sb_iommu == NULL) > > > - panic("iommu_dvmamap_sync: null iommu"); > > > -#endif > > > if (len == 0) > > > return; > > > > > > + if (map->_dm_flags & BUS_DMA_64BIT) { > > > + if (ops & (BUS_DMASYNC_PREWRITE | BUS_DMASYNC_POSTREAD)) > > > + membar(MemIssue); > > > + return; > > > + } > > > + > > > + ims = map->_dm_cookie; > > > + > > > if (ops & BUS_DMASYNC_PREWRITE) > > > membar(MemIssue); > > > > > > @@ -1622,9 +1630,13 @@ iommu_dvmamem_alloc(bus_dma_tag_t t, bus > > > "bound %llx segp %p flags %d\n", (unsigned long long)size, > > > (unsigned long long)alignment, (unsigned long long)boundary, > > > segs, flags)); > > > + > > > + if ((flags & BUS_DMA_64BIT) == 0) > > > + flags |= BUS_DMA_DVMA; > > > + > > > BUS_DMA_FIND_PARENT(t, _dmamem_alloc); > > > return ((*t->_dmamem_alloc)(t, t0, size, alignment, boundary, > > > - segs, nsegs, rsegs, flags | BUS_DMA_DVMA)); > > > + segs, nsegs, rsegs, flags)); > > > } > > > > > > void > > > @@ -1763,7 +1775,7 @@ iommu_iomap_load_map(struct iommu_state > > > > > > /* Flush cache if necessary. */ > > > slot = IOTSBSLOT(e->ipe_va, is->is_tsbsize); > > > - if (is->is_flags & IOMMU_FLUSH_CACHE && > > > + if (ISSET(is->is_hw->ihw_flags, IOMMU_HW_FLUSH_CACHE) && > > > (i == (ipm->ipm_pagecnt - 1) || (slot % 8) == 7)) > > > IOMMUREG_WRITE(is, iommu_cache_flush, > > > is->is_ptsb + slot * 8); > > > @@ -1788,7 +1800,7 @@ iommu_iomap_unload_map(struct iommu_stat > > > > > > /* Flush cache if necessary. */ > > > slot = IOTSBSLOT(e->ipe_va, is->is_tsbsize); > > > - if (is->is_flags & IOMMU_FLUSH_CACHE && > > > + if (ISSET(is->is_hw->ihw_flags, IOMMU_HW_FLUSH_CACHE) && > > > (i == (ipm->ipm_pagecnt - 1) || (slot % 8) == 7)) > > > IOMMUREG_WRITE(is, iommu_cache_flush, > > > is->is_ptsb + slot * 8); > > > Index: dev/iommureg.h > > > =================================================================== > > > RCS file: /cvs/src/sys/arch/sparc64/dev/iommureg.h,v > > > retrieving revision 1.17 > > > diff -u -p -r1.17 iommureg.h > > > --- dev/iommureg.h 17 Aug 2012 20:46:50 -0000 1.17 > > > +++ dev/iommureg.h 10 May 2017 12:00:09 -0000 > > > @@ -90,10 +90,11 @@ struct iommu_strbuf { > > > #define IOMMUCR_DE 0x000000000000000002LL /* Diag enable */ > > > #define IOMMUCR_EN 0x000000000000000001LL /* Enable IOMMU */ > > > > > > -#define IOMMUCR_FIRE_SE 0x000000000000000400LL /* Snoop > enable */ > > > -#define IOMMUCR_FIRE_CM_EN 0x000000000000000300LL /* Cache mode > enable */ > > > -#define IOMMUCR_FIRE_BE 0x000000000000000002LL /* Bypass > enable */ > > > -#define IOMMUCR_FIRE_TE 0x000000000000000001LL /* > Translation enabled */ > > > +#define IOMMUCR_FIRE_PD 0x000000000000001000UL /* Process > disable */ > > > +#define IOMMUCR_FIRE_SE 0x000000000000000400UL /* Snoop > enable */ > > > +#define IOMMUCR_FIRE_CM_EN 0x000000000000000300UL /* Cache mode > enable */ > > > +#define IOMMUCR_FIRE_BE 0x000000000000000002UL /* Bypass > enable */ > > > +#define IOMMUCR_FIRE_TE 0x000000000000000001UL /* > Translation enabled */ > > > > > > /* > > > * IOMMU stuff > > > Index: dev/iommuvar.h > > > =================================================================== > > > RCS file: /cvs/src/sys/arch/sparc64/dev/iommuvar.h,v > > > retrieving revision 1.17 > > > diff -u -p -r1.17 iommuvar.h > > > --- dev/iommuvar.h 4 May 2016 18:26:12 -0000 1.17 > > > +++ dev/iommuvar.h 10 May 2017 12:00:09 -0000 > > > @@ -100,6 +100,21 @@ struct iommu_map_state { > > > }; > > > #define IOMMU_MAP_STREAM 1 > > > > > > +struct iommu_hw { > > > + void (*ihw_enable)(struct iommu_state *); > > > + > > > + unsigned long ihw_dvma_pa; > > > + > > > + unsigned long ihw_bypass; > > > + unsigned long ihw_bypass_nc; /* non-cached */ > > > + unsigned long ihw_bypass_ro; /* relaxed > ordering */ > > > + > > > + unsigned int ihw_flags; > > > +#define IOMMU_HW_FLUSH_CACHE (1 << 0) > > > +}; > > > + > > > +extern const struct iommu_hw iommu_hw_default; > > > + > > > /* > > > * per-IOMMU state > > > */ > > > @@ -112,8 +127,7 @@ struct iommu_state { > > > int64_t is_cr; /* Control register value > */ > > > struct mutex is_mtx; > > > struct extent *is_dvmamap; /* DVMA map for this > instance */ > > > - int is_flags; > > > -#define IOMMU_FLUSH_CACHE 0x00000001 > > > + const struct iommu_hw *is_hw; > > > > > > struct strbuf_ctl *is_sb[2]; /* Streaming buffers if > any */ > > > > > > @@ -126,7 +140,8 @@ struct iommu_state { > > > }; > > > > > > /* interfaces for PCI/SBus code */ > > > -void iommu_init(char *, struct iommu_state *, int, u_int32_t); > > > +void iommu_init(char *, const struct iommu_hw *, struct > iommu_state *, > > > + int, u_int32_t); > > > void iommu_reset(struct iommu_state *); > > > paddr_t iommu_extract(struct iommu_state *, bus_addr_t); > > > int64_t iommu_lookup_tte(struct iommu_state *, bus_addr_t); > > > @@ -146,6 +161,7 @@ int iommu_dvmamem_alloc(bus_dma_tag_t, b > > > bus_size_t, bus_size_t, bus_dma_segment_t *, int, int *, int); > > > void iommu_dvmamem_free(bus_dma_tag_t, bus_dma_tag_t, > bus_dma_segment_t *, > > > int); > > > + > > > > > > #define IOMMUREG_READ(is, reg) \ > > > bus_space_read_8((is)->is_bustag, \ > > > Index: dev/pci_machdep.c > > > =================================================================== > > > RCS file: /cvs/src/sys/arch/sparc64/dev/pci_machdep.c,v > > > retrieving revision 1.44 > > > diff -u -p -r1.44 pci_machdep.c > > > --- dev/pci_machdep.c 10 May 2014 12:15:19 -0000 1.44 > > > +++ dev/pci_machdep.c 10 May 2017 12:00:09 -0000 > > > @@ -58,6 +58,7 @@ int sparc_pci_debug = 0x0; > > > #include <machine/openfirm.h> > > > #include <dev/pci/pcivar.h> > > > #include <dev/pci/pcireg.h> > > > +#include <dev/pci/pcidevs.h> > > > > > > #include <dev/ofw/ofw_pci.h> > > > > > > @@ -85,6 +86,46 @@ pci_attach_hook(parent, self, pba) > > > struct pcibus_attach_args *pba; > > > { > > > /* Don't do anything */ > > > +} > > > + > > > +int > > > +pci_bcm_dmamap_create(bus_dma_tag_t dt, bus_dma_tag_t t0, bus_size_t > size, > > > + int nsegments, bus_size_t maxsegsz, bus_size_t boundary, int > flags, > > > + bus_dmamap_t *dmamp) > > > +{ > > > + bus_dma_tag_t pdt = dt->_parent; > > > + > > > + CLR(flags, BUS_DMA_64BIT); > > > + > > > + return ((*pdt->_dmamap_create)(pdt, t0, size, nsegments, maxsegsz, > > > + boundary, flags, dmamp)); > > > +} > > > + > > > +int > > > +pci_probe_device_hook(pci_chipset_tag_t pc, struct pci_attach_args > *pa) > > > +{ > > > + bus_dma_tag_t dt, pdt; > > > + > > > + if (pa->pa_id == > > > + PCI_ID_CODE(PCI_VENDOR_RCC, PCI_PRODUCT_RCC_PCIE_PCIX)) { > > > + /* > > > + * These PCI bridges only support 40bit DVA, so intercept > > > + * bus_dmamap_create so we can clear BUS_DMA_64BIT. > > > + */ > > > + > > > + dt = malloc(sizeof(*dt), M_DEVBUF, M_NOWAIT | M_ZERO); > > > + if (dt == NULL) > > > + panic("%s: could not alloc dma tag", __func__); > > > + > > > + pdt = pa->pa_dmat; > > > + > > > + dt->_parent = pdt; > > > + dt->_dmamap_create = pci_bcm_dmamap_create; > > > + > > > + pa->pa_dmat = dt; > > > + } > > > + > > > + return (0); > > > } > > > > > > int > > > Index: dev/psycho.c > > > =================================================================== > > > RCS file: /cvs/src/sys/arch/sparc64/dev/psycho.c,v > > > retrieving revision 1.74 > > > diff -u -p -r1.74 psycho.c > > > --- dev/psycho.c 23 Aug 2016 03:28:01 -0000 1.74 > > > +++ dev/psycho.c 10 May 2017 12:00:09 -0000 > > > @@ -902,7 +902,7 @@ psycho_iommu_init(struct psycho_softc *s > > > panic("couldn't malloc iommu name"); > > > snprintf(name, 32, "%s dvma", sc->sc_dev.dv_xname); > > > > > > - iommu_init(name, is, tsbsize, iobase); > > > + iommu_init(name, &iommu_hw_default, is, tsbsize, iobase); > > > } > > > > > > /* > > > Index: dev/pyro.c > > > =================================================================== > > > RCS file: /cvs/src/sys/arch/sparc64/dev/pyro.c,v > > > retrieving revision 1.30 > > > diff -u -p -r1.30 pyro.c > > > --- dev/pyro.c 20 Dec 2016 13:40:50 -0000 1.30 > > > +++ dev/pyro.c 10 May 2017 12:00:09 -0000 > > > @@ -131,6 +131,30 @@ int pyro_msi_eq_intr(void *); > > > int pyro_dmamap_create(bus_dma_tag_t, bus_dma_tag_t, bus_size_t, int, > > > bus_size_t, bus_size_t, int, bus_dmamap_t *); > > > > > > +void pyro_iommu_enable(struct iommu_state *); > > > + > > > +const struct iommu_hw iommu_hw_fire = { > > > + .ihw_enable = pyro_iommu_enable, > > > + > > > + .ihw_dvma_pa = 0x000007ffffffffffUL, > > > + > > > + .ihw_bypass = 0xfffc000000000000UL, > > > + .ihw_bypass_nc = 0x0000080000000000UL, > > > + .ihw_bypass_ro = 0, > > > +}; > > > + > > > +const struct iommu_hw iommu_hw_oberon = { > > > + .ihw_enable = pyro_iommu_enable, > > > + > > > + .ihw_dvma_pa = 0x00007fffffffffffUL, > > > + > > > + .ihw_bypass = 0x7ffc000000000000UL, > > > + .ihw_bypass_nc = 0x0000800000000000UL, > > > + .ihw_bypass_ro = 0x8000000000000000UL, > > > + > > > + .ihw_flags = IOMMU_HW_FLUSH_CACHE, > > > +}; > > > + > > > #ifdef DDB > > > void pyro_xir(void *, int); > > > #endif > > > @@ -266,6 +290,7 @@ pyro_init_iommu(struct pyro_softc *sc, s > > > int tsbsize = 7; > > > u_int32_t iobase = -1; > > > char *name; > > > + const struct iommu_hw *ihw = &iommu_hw_fire; > > > > > > is->is_bustag = sc->sc_bust; > > > > > > @@ -282,11 +307,23 @@ pyro_init_iommu(struct pyro_softc *sc, s > > > panic("couldn't malloc iommu name"); > > > snprintf(name, 32, "%s dvma", sc->sc_dv.dv_xname); > > > > > > - /* On Oberon, we need to flush the cache. */ > > > if (sc->sc_oberon) > > > - is->is_flags |= IOMMU_FLUSH_CACHE; > > > + ihw = &iommu_hw_oberon; > > > + > > > + iommu_init(name, ihw, is, tsbsize, iobase); > > > +} > > > + > > > +void > > > +pyro_iommu_enable(struct iommu_state *is) > > > +{ > > > + unsigned long cr; > > > + > > > + cr = IOMMUREG_READ(is, iommu_cr); > > > + cr |= IOMMUCR_FIRE_BE | IOMMUCR_FIRE_SE | IOMMUCR_FIRE_CM_EN | > > > + IOMMUCR_FIRE_TE; > > > > > > - iommu_init(name, is, tsbsize, iobase); > > > + IOMMUREG_WRITE(is, iommu_tsb, is->is_ptsb | is->is_tsbsize); > > > + IOMMUREG_WRITE(is, iommu_cr, cr); > > > } > > > > > > void > > > Index: dev/sbus.c > > > =================================================================== > > > RCS file: /cvs/src/sys/arch/sparc64/dev/sbus.c,v > > > retrieving revision 1.44 > > > diff -u -p -r1.44 sbus.c > > > --- dev/sbus.c 19 Sep 2015 21:07:04 -0000 1.44 > > > +++ dev/sbus.c 10 May 2017 12:00:09 -0000 > > > @@ -349,7 +349,7 @@ sbus_mb_attach(struct device *parent, st > > > snprintf(name, 32, "%s dvma", sc->sc_dev.dv_xname); > > > > > > printf("%s: ", sc->sc_dev.dv_xname); > > > - iommu_init(name, &sc->sc_is, 0, -1); > > > + iommu_init(name, &iommu_hw_default, &sc->sc_is, 0, -1); > > > > > > /* Initialize Starfire PC interrupt translation. */ > > > if (OF_getprop(findroot(), "name", buf, sizeof(buf)) > 0 && > > > Index: dev/schizo.c > > > =================================================================== > > > RCS file: /cvs/src/sys/arch/sparc64/dev/schizo.c,v > > > retrieving revision 1.67 > > > diff -u -p -r1.67 schizo.c > > > --- dev/schizo.c 23 Aug 2016 03:28:01 -0000 1.67 > > > +++ dev/schizo.c 10 May 2017 12:00:09 -0000 > > > @@ -451,7 +451,7 @@ schizo_init_iommu(struct schizo_softc *s > > > "using iobase=0x%x, tsbsize=%d\n", iobase, tsbsize)); > > > } > > > > > > - iommu_init(name, is, tsbsize, iobase); > > > + iommu_init(name, &iommu_hw_default, is, tsbsize, iobase); > > > } > > > > > > int > > > Index: include/pci_machdep.h > > > =================================================================== > > > RCS file: /cvs/src/sys/arch/sparc64/include/pci_machdep.h,v > > > retrieving revision 1.33 > > > diff -u -p -r1.33 pci_machdep.h > > > --- include/pci_machdep.h 4 May 2016 14:30:01 -0000 1.33 > > > +++ include/pci_machdep.h 10 May 2017 12:00:09 -0000 > > > @@ -74,10 +74,13 @@ struct sparc_pci_chipset { > > > pcireg_t (*conf_read)(pci_chipset_tag_t, pcitag_t, int); > > > void (*conf_write)(pci_chipset_tag_t, pcitag_t, int, pcireg_t); > > > int (*intr_map)(struct pci_attach_args *, pci_intr_handle_t *); > > > + int (*probe_device_hook)(void *, struct pci_attach_args *); > > > }; > > > > > > void pci_attach_hook(struct device *, struct device *, > > > struct pcibus_attach_args *); > > > +int pci_probe_device_hook(pci_chipset_tag_t, > > > + struct pci_attach_args *); > > > int pci_bus_maxdevs(pci_chipset_tag_t, int); > > > pcitag_t pci_make_tag(pci_chipset_tag_t, int, int, int); > > > void pci_decompose_tag(pci_chipset_tag_t, pcitag_t, int > *, int *, > > > @@ -102,8 +105,6 @@ int sparc64_pci_enumerate_bus(struct pc > > > struct pci_attach_args *); > > > > > > #define PCI_MACHDEP_ENUMERATE_BUS sparc64_pci_enumerate_bus > > > - > > > -#define pci_probe_device_hook(c, a) (0) > > > > > > #define pci_min_powerstate(c, t) (PCI_PMCSR_STATE_D3) > > > #define pci_set_powerstate_md(c, t, s, p) > > > > > >
