Re: [net-next PATCH v1 0/2] stmmac: dwmac-meson8b: configurable RGMII TX delay
On 25/11/16 18:44, Florian Fainelli wrote: > On 11/25/2016 03:13 AM, Sebastian Frias wrote: >> On 24/11/16 19:55, Florian Fainelli wrote: >>> Correct, the meaning of PHY_INTERFACE_MODE should be from the >>> perspective of the PHY device: >>> >>> - PHY_INTERFACE_MODE_RGMII_TXID means that the PHY is responsible for >>> adding a delay when the MAC transmits (TX MAC -> PHY (delay) -> wire) >>> - PHY_INTERFACE_MODE_RGMII_RXID means that the PHY is responsible for >>> adding a delay when the MAC receives (RX MAC <- (delay) PHY) <- wire) >>> >> >> Thanks for the explanation. >> Actually I had thought that the delay was to account for board routing >> (wires) between the MAC and the PHY. >> From your explanation it appears that the delay is to account for board >> routing (wires) between the PHY and the RJ45 socket. > > The placement of the (delay) was not meant to be exact, but it was > wrongly place anyway, so it should be between the MAC and PHY, always. > This is why you see people either fixing the need for a delay by > appropriately programming the PHY, or the MAC, or by just inserting a > fixed delay on the PCB between the PHY and the MAC and programming no > delays (or using the default values and hoping this works). Thanks. Your patch "[PATCH net-next 3/4] Documentation: net: phy: Add blurb about RGMII" on the documentation makes it clear. >>> >>> This also seems reasonable to do, provided that the PHY is also properly >>> configured not to add delays in both directions, and therefore assumes >>> that the MAC does it. >>> >>> We have a fairly large problem with how RGMII delays are done in PHYLIB >>> and Ethernet MAC drivers (or just in general), where we can't really >>> intersect properly what a PHY is supporting (in terms of internal >>> delays), and what the MAC supports either. One possible approach could >>> be to update PHY drivers a list of PHY_INTERFACE_MODE_* that they >>> support (ideally, even with normalized nanosecond delay values), >> >> Just to make sure I understood this, the DT would say something like: >> >> phy-connection-type = "rgmii-txid"; >> txid-delay-ns = <3>; >> >> For a 3ns TX delay, would that be good? > > That's one possibility, although, see below, some PHYs support > sub-nanosecond values, but in general, that seems like a good > representation. If the "txid-delay-ns" property is omitted, a standard > 2ns delay is assumed. Sounds good. I did not see the "txid-delay-ns" property documented in your patches, if it is not too late, maybe it could be "txid-delay-ps" using picoseconds as unit, right? >>> and >>> then intersect that with the requested phy_interface_t during >>> phy_{attach,connect} time, and feed this back to the MAC with a special >>> error code/callback, so we could gracefully try to choose another >>> PHY_INTERFACE_MODE_* value that the MAC supports >>> >>> A larger problem is that a number of drivers have been deployed, and >>> Device Trees, possibly with the meaning of "phy-mode" and >>> "phy-connection-type" being from the MAC perspective, and not the PHY >>> perspective *sigh*, good luck auditing those. >>> >>> So from there, here is possibly what we could do >>> >>> - submit a series of patches that update the PHYLIB documentation (there >>> are other things missing here) and make it clear from which entity (PHY >>> or MAC) does the delay apply to, document the "intersection" problem here >> >> I think documenting is necessary, thanks in advance! >> >> However, I'm wondering if there's a way to make this work in all cases. >> Indeed, if we consider for example that TX delay is required, we have 4 >> cases: >> >>PHY | MAC | Who applies? >> TXID supported | TXID supported | PHY >> TXID supported | TXID not supported | PHY >> TXID not supported | TXID supported | MAC >> TXID not supported | TXID not supported | cannot be done >> >> That is basically what my patch: >> >> https://marc.info/?l=linux-netdev=147869658031783=2 >> >> attempted to achieve. That would allow more combinations of MAC<->PHY to >> work, right? > > Yes, indeed. Just one thing, from your patch "[PATCH net-next 3/4] Documentation: net: phy: Add blurb about RGMII" I have the impression that the 3rd option from the table above, would be a little bit more complex to implement. I will comment on the patch. >> Nevertheless, I think we also need to keep in mind that most of this >> discussion assumes the case where both, MAC and PHY have equal capabilities. >> Could it happen that the PHY supports only 2ns delay, and the MAC only >> 1ns delay? > > I doubt this exists at the MAC level what we should have is either a 2ns > delay, in either RX or TX path, or nothing, because that's the value > that results in shifting the data lines and the RX/TX lines by 90 > degrees at 125Mhz (1/125^6 = 8 ns, one quarter shift is 90 degrees = > 2ns). The PHY may have a similar set of pre-programmed, fixed 2ns > delays, but it is not uncommon to see
Re: [net-next PATCH v1 0/2] stmmac: dwmac-meson8b: configurable RGMII TX delay
On 11/25/2016 03:13 AM, Sebastian Frias wrote: > On 24/11/16 19:55, Florian Fainelli wrote: >> Le 24/11/2016 à 09:05, Martin Blumenstingl a écrit : >>> On Thu, Nov 24, 2016 at 4:56 PM, Jerome Brunetwrote: On Thu, 2016-11-24 at 15:34 +0100, Martin Blumenstingl wrote: > Currently the dwmac-meson8b stmmac glue driver uses a hardcoded 1/4 > cycle TX clock delay. This seems to work fine for many boards (for > example Odroid-C2 or Amlogic's reference boards) but there are some > others where TX traffic is simply broken. > There are probably multiple reasons why it's working on some boards > while it's broken on others: > - some of Amlogic's reference boards are using a Micrel PHY > - hardware circuit design > - maybe more... > > This raises a question though: > Which device is supposed to enable the TX delay when both MAC and PHY > support it? And should we implement it for each PHY / MAC separately > or should we think about a more generic solution (currently it's not > possible to disable the TX delay generated by the RTL8211F PHY via > devicetree when using phy-mode "rgmii")? Actually you can skip the part which activate the Tx-delay on the phy by setting "phy-mode = "rgmii-id" instead of "rgmii" phy->interface will no longer be PHY_INTERFACE_MODE_RGMII but PHY_INTERFACE_MODE_RGMII_ID. >>> unfortunately this is not true for RTL8211F (I did my previous tests >>> with the same expectation in mind)! >>> the code seems to suggest that TX-delay is disabled whenever mode != >>> PHY_INTERFACE_MODE_RGMII. >>> BUT: on my device RTL8211F_TX_DELAY is set even before >>> "phy_write(phydev, 0x11, reg);"! > > If you look at the Atheros 803x PHY and its driver > 'drivers/net/phy/at803x.c': > - by default (as HW reset preset) the PHY has RX delay enabled, TX > delay disabled > - the driver only enables RX, or TX, or both, according to "rgmii-rxid", > "rgmii-txid", or "rgmii-id" respectively, but does not alter HW reset > presets. In other words: > a "rgmii-rxid" results in RX enabled (expected) > b "rgmii-txid" results in RX *and* TX enabled (unexpected?) > c "rgmii-id" results in RX *and* TX enabled (expected) > d "rgmii" results in RX enabled (unexpected?) > > This is a bit surprising and I think that some boards and PHY<->MAC > combinations are working a little bit by chance, unless I'm missing > something. > >> >> (Adding Sebastian (and Mans, and Andrew) since he raised the same >> question a while ago. I think I now understand a bit better what >> Sebastian was after a couple of weeks ago) >> > > Thanks for CCing us, it is indeed a very similar issue. > >>> >>> Based on what I found it seems that rgmii-id, rgmii-txid and >>> rgmii-rxid are supposed to be handled by the PHY. >> >> Correct, the meaning of PHY_INTERFACE_MODE should be from the >> perspective of the PHY device: >> >> - PHY_INTERFACE_MODE_RGMII_TXID means that the PHY is responsible for >> adding a delay when the MAC transmits (TX MAC -> PHY (delay) -> wire) >> - PHY_INTERFACE_MODE_RGMII_RXID means that the PHY is responsible for >> adding a delay when the MAC receives (RX MAC <- (delay) PHY) <- wire) >> > > Thanks for the explanation. > Actually I had thought that the delay was to account for board routing > (wires) between the MAC and the PHY. > From your explanation it appears that the delay is to account for board > routing (wires) between the PHY and the RJ45 socket. The placement of the (delay) was not meant to be exact, but it was wrongly place anyway, so it should be between the MAC and PHY, always. This is why you see people either fixing the need for a delay by appropriately programming the PHY, or the MAC, or by just inserting a fixed delay on the PCB between the PHY and the MAC and programming no delays (or using the default values and hoping this works). > >>> That would mean that we have two problems here: >>> 1) drivers/net/phy/realtek.c:rtl8211f_config_init should check for >>> PHY_INTERFACE_MODE_RGMII_ID or PHY_INTERFACE_MODE_RGMII_TXID and >>> enable the TX-delay in that case - otherwise explicitly disable it >> >> Agreed. >> >>> 2) dwmac-meson8b.c should only use the configured TX-delay for >>> PHY_INTERFACE_MODE_RGMII >>> @Florian: could you please share your thoughts on this (who handles >>> the TX delay in which case)? >> >> This also seems reasonable to do, provided that the PHY is also properly >> configured not to add delays in both directions, and therefore assumes >> that the MAC does it. >> >> We have a fairly large problem with how RGMII delays are done in PHYLIB >> and Ethernet MAC drivers (or just in general), where we can't really >> intersect properly what a PHY is supporting (in terms of internal >> delays), and what the MAC supports either. One possible approach could >> be to update PHY drivers a list of PHY_INTERFACE_MODE_* that they >> support (ideally, even with normalized
Re: [net-next PATCH v1 0/2] stmmac: dwmac-meson8b: configurable RGMII TX delay
Sebastian Friaswrites: > On 24/11/16 19:55, Florian Fainelli wrote: >> Le 24/11/2016 à 09:05, Martin Blumenstingl a écrit : >>> Based on what I found it seems that rgmii-id, rgmii-txid and >>> rgmii-rxid are supposed to be handled by the PHY. >> >> Correct, the meaning of PHY_INTERFACE_MODE should be from the >> perspective of the PHY device: >> >> - PHY_INTERFACE_MODE_RGMII_TXID means that the PHY is responsible for >> adding a delay when the MAC transmits (TX MAC -> PHY (delay) -> wire) >> - PHY_INTERFACE_MODE_RGMII_RXID means that the PHY is responsible for >> adding a delay when the MAC receives (RX MAC <- (delay) PHY) <- wire) >> > > Thanks for the explanation. > Actually I had thought that the delay was to account for board routing > (wires) between the MAC and the PHY. > From your explanation it appears that the delay is to account for board > routing (wires) between the PHY and the RJ45 socket. The delay pertains to the RGMII link between MAC and PHY. The external connection is self-clocking. -- Måns Rullgård
Re: [net-next PATCH v1 0/2] stmmac: dwmac-meson8b: configurable RGMII TX delay
On 24/11/16 19:55, Florian Fainelli wrote: > Le 24/11/2016 à 09:05, Martin Blumenstingl a écrit : >> On Thu, Nov 24, 2016 at 4:56 PM, Jerome Brunetwrote: >>> On Thu, 2016-11-24 at 15:34 +0100, Martin Blumenstingl wrote: Currently the dwmac-meson8b stmmac glue driver uses a hardcoded 1/4 cycle TX clock delay. This seems to work fine for many boards (for example Odroid-C2 or Amlogic's reference boards) but there are some others where TX traffic is simply broken. There are probably multiple reasons why it's working on some boards while it's broken on others: - some of Amlogic's reference boards are using a Micrel PHY - hardware circuit design - maybe more... This raises a question though: Which device is supposed to enable the TX delay when both MAC and PHY support it? And should we implement it for each PHY / MAC separately or should we think about a more generic solution (currently it's not possible to disable the TX delay generated by the RTL8211F PHY via devicetree when using phy-mode "rgmii")? >>> >>> Actually you can skip the part which activate the Tx-delay on the phy >>> by setting "phy-mode = "rgmii-id" instead of "rgmii" >>> >>> phy->interface will no longer be PHY_INTERFACE_MODE_RGMII >>> but PHY_INTERFACE_MODE_RGMII_ID. >> unfortunately this is not true for RTL8211F (I did my previous tests >> with the same expectation in mind)! >> the code seems to suggest that TX-delay is disabled whenever mode != >> PHY_INTERFACE_MODE_RGMII. >> BUT: on my device RTL8211F_TX_DELAY is set even before >> "phy_write(phydev, 0x11, reg);"! If you look at the Atheros 803x PHY and its driver 'drivers/net/phy/at803x.c': - by default (as HW reset preset) the PHY has RX delay enabled, TX delay disabled - the driver only enables RX, or TX, or both, according to "rgmii-rxid", "rgmii-txid", or "rgmii-id" respectively, but does not alter HW reset presets. In other words: a "rgmii-rxid" results in RX enabled (expected) b "rgmii-txid" results in RX *and* TX enabled (unexpected?) c "rgmii-id" results in RX *and* TX enabled (expected) d "rgmii" results in RX enabled (unexpected?) This is a bit surprising and I think that some boards and PHY<->MAC combinations are working a little bit by chance, unless I'm missing something. > > (Adding Sebastian (and Mans, and Andrew) since he raised the same > question a while ago. I think I now understand a bit better what > Sebastian was after a couple of weeks ago) > Thanks for CCing us, it is indeed a very similar issue. >> >> Based on what I found it seems that rgmii-id, rgmii-txid and >> rgmii-rxid are supposed to be handled by the PHY. > > Correct, the meaning of PHY_INTERFACE_MODE should be from the > perspective of the PHY device: > > - PHY_INTERFACE_MODE_RGMII_TXID means that the PHY is responsible for > adding a delay when the MAC transmits (TX MAC -> PHY (delay) -> wire) > - PHY_INTERFACE_MODE_RGMII_RXID means that the PHY is responsible for > adding a delay when the MAC receives (RX MAC <- (delay) PHY) <- wire) > Thanks for the explanation. Actually I had thought that the delay was to account for board routing (wires) between the MAC and the PHY. >From your explanation it appears that the delay is to account for board routing (wires) between the PHY and the RJ45 socket. >> That would mean that we have two problems here: >> 1) drivers/net/phy/realtek.c:rtl8211f_config_init should check for >> PHY_INTERFACE_MODE_RGMII_ID or PHY_INTERFACE_MODE_RGMII_TXID and >> enable the TX-delay in that case - otherwise explicitly disable it > > Agreed. > >> 2) dwmac-meson8b.c should only use the configured TX-delay for >> PHY_INTERFACE_MODE_RGMII >> @Florian: could you please share your thoughts on this (who handles >> the TX delay in which case)? > > This also seems reasonable to do, provided that the PHY is also properly > configured not to add delays in both directions, and therefore assumes > that the MAC does it. > > We have a fairly large problem with how RGMII delays are done in PHYLIB > and Ethernet MAC drivers (or just in general), where we can't really > intersect properly what a PHY is supporting (in terms of internal > delays), and what the MAC supports either. One possible approach could > be to update PHY drivers a list of PHY_INTERFACE_MODE_* that they > support (ideally, even with normalized nanosecond delay values), Just to make sure I understood this, the DT would say something like: phy-connection-type = "rgmii-txid"; txid-delay-ns = <3>; For a 3ns TX delay, would that be good? >and > then intersect that with the requested phy_interface_t during > phy_{attach,connect} time, and feed this back to the MAC with a special > error code/callback, so we could gracefully try to choose another > PHY_INTERFACE_MODE_* value that the MAC supports > > A larger problem is that a number of drivers have been deployed, and > Device Trees, possibly with
Re: [net-next PATCH v1 0/2] stmmac: dwmac-meson8b: configurable RGMII TX delay
On Thu, 2016-11-24 at 18:05 +0100, Martin Blumenstingl wrote: > On Thu, Nov 24, 2016 at 4:56 PM, Jerome Brunet> wrote: > > > > On Thu, 2016-11-24 at 15:34 +0100, Martin Blumenstingl wrote: > > > > > > Currently the dwmac-meson8b stmmac glue driver uses a hardcoded > > > 1/4 > > > cycle TX clock delay. This seems to work fine for many boards > > > (for > > > example Odroid-C2 or Amlogic's reference boards) but there are > > > some > > > others where TX traffic is simply broken. > > > There are probably multiple reasons why it's working on some > > > boards > > > while it's broken on others: > > > - some of Amlogic's reference boards are using a Micrel PHY > > > - hardware circuit design > > > - maybe more... > > > > > > This raises a question though: > > > Which device is supposed to enable the TX delay when both MAC and > > > PHY > > > support it? And should we implement it for each PHY / MAC > > > separately > > > or should we think about a more generic solution (currently it's > > > not > > > possible to disable the TX delay generated by the RTL8211F PHY > > > via > > > devicetree when using phy-mode "rgmii")? > > > > Actually you can skip the part which activate the Tx-delay on the > > phy > > by setting "phy-mode = "rgmii-id" instead of "rgmii" > > > > phy->interface will no longer be PHY_INTERFACE_MODE_RGMII > > but PHY_INTERFACE_MODE_RGMII_ID. > unfortunately this is not true for RTL8211F (I did my previous tests > with the same expectation in mind)! > the code seems to suggest that TX-delay is disabled whenever mode != > PHY_INTERFACE_MODE_RGMII. > BUT: on my device RTL8211F_TX_DELAY is set even before > "phy_write(phydev, 0x11, reg);"! Thx a lot for pointing this out. I wrongly assumed that is was off by default. It turns out it is ON on the OdroidC2 as well. So I re-did all my test: On the OdroidC2, as long as eee is not disabled (using ethtool or through my patch) I can't get iperf to work properly. Disabling EEE, the link works for the following configurations: PHY TX Delay bit set : MAC delay 0ns, 2ns. PHY TX Delay bit cleared : MAC delay 2ns, 4ns. This seems to confirm that the PHY adds a 2ns delay when the bit is set. I could not test on my MXQPRO2.1 ... the PHY is no longer found during the boot. Something died overnight :( > > Based on what I found it seems that rgmii-id, rgmii-txid and > rgmii-rxid are supposed to be handled by the PHY. > That would mean that we have two problems here: > 1) drivers/net/phy/realtek.c:rtl8211f_config_init should check for > PHY_INTERFACE_MODE_RGMII_ID or PHY_INTERFACE_MODE_RGMII_TXID and > enable the TX-delay in that case - otherwise explicitly disable it > 2) dwmac-meson8b.c should only use the configured TX-delay for > PHY_INTERFACE_MODE_RGMII > @Florian: could you please share your thoughts on this (who handles > the TX delay in which case)? > > > Regards, > Martin
Re: [net-next PATCH v1 0/2] stmmac: dwmac-meson8b: configurable RGMII TX delay
On 11/24/2016 5:08 PM, Jerome Brunet wrote: On Thu, 2016-11-24 at 15:34 +0100, Martin Blumenstingl wrote: Currently the dwmac-meson8b stmmac glue driver uses a hardcoded 1/4 cycle TX clock delay. This seems to work fine for many boards (for example Odroid-C2 or Amlogic's reference boards) but there are some others where TX traffic is simply broken. There are probably multiple reasons why it's working on some boards while it's broken on others: - some of Amlogic's reference boards are using a Micrel PHY - hardware circuit design - maybe more... This raises a question though: Which device is supposed to enable the TX delay when both MAC and PHY support it? And should we implement it for each PHY / MAC separately or should we think about a more generic solution (currently it's not possible to disable the TX delay generated by the RTL8211F PHY via devicetree when using phy-mode "rgmii")? iperf3 results on my Mecool BB2 board (Meson GXM, RTL8211F PHY) with TX clock delay disabled on the MAC (as it's enabled in the PHY driver). TX throughput was virtually zero before: $ iperf3 -c 192.168.1.100 -R Connecting to host 192.168.1.100, port 5201 Reverse mode, remote host 192.168.1.100 is sending [ 4] local 192.168.1.206 port 52828 connected to 192.168.1.100 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 108 MBytes 901 Mbits/sec [ 4] 1.00-2.00 sec 94.2 MBytes 791 Mbits/sec [ 4] 2.00-3.00 sec 96.5 MBytes 810 Mbits/sec [ 4] 3.00-4.00 sec 96.2 MBytes 808 Mbits/sec [ 4] 4.00-5.00 sec 96.6 MBytes 810 Mbits/sec [ 4] 5.00-6.00 sec 96.5 MBytes 810 Mbits/sec [ 4] 6.00-7.00 sec 96.6 MBytes 810 Mbits/sec [ 4] 7.00-8.00 sec 96.5 MBytes 809 Mbits/sec [ 4] 8.00-9.00 sec 105 MBytes 884 Mbits/sec [ 4] 9.00-10.00 sec 111 MBytes 934 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 1000 MBytes 839 Mbits/sec0 sender [ 4] 0.00-10.00 sec 998 MBytes 837 Mbits/sec receiver iperf Done. $ iperf3 -c 192.168.1.100 Connecting to host 192.168.1.100, port 5201 [ 4] local 192.168.1.206 port 52832 connected to 192.168.1.100 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.01 sec 99.5 MBytes 829 Mbits/sec 117139 KBytes [ 4] 1.01-2.00 sec 105 MBytes 884 Mbits/sec 129 70.7 KBytes [ 4] 2.00-3.01 sec 107 MBytes 889 Mbits/sec 106187 KBytes [ 4] 3.01-4.01 sec 105 MBytes 878 Mbits/sec 92143 KBytes [ 4] 4.01-5.00 sec 105 MBytes 882 Mbits/sec 140129 KBytes [ 4] 5.00-6.01 sec 106 MBytes 883 Mbits/sec 115195 KBytes [ 4] 6.01-7.00 sec 102 MBytes 863 Mbits/sec 133 70.7 KBytes [ 4] 7.00-8.01 sec 106 MBytes 884 Mbits/sec 143 97.6 KBytes [ 4] 8.01-9.01 sec 104 MBytes 875 Mbits/sec 124107 KBytes [ 4] 9.01-10.01 sec 105 MBytes 876 Mbits/sec 90139 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.01 sec 1.02 GBytes 874 Mbits/sec 1189 sender [ 4] 0.00-10.01 sec 1.02 GBytes 873 Mbits/sec receiver Cool, one more board working ;) I tried your patch on another board (MXQ_V2.1, with sheep printed on the PCB). It 's not improving the situation for this one unfortunately. Actually I already tried playing with the TX delay on the MAC and PHY but I could get any good results with the boards I have. It is strange that we can adjust the delay by 2ns steps, when delay seens by the phy should be 2ns ... Ok, as said, I could expect that extra delay was needed on GiGa. FYI, on ST platforms, this extra delay was added in the pintctrl dtsi. I mean, in some way, ad-hoc setup was solved at device-tree level. Usually, extra delay could depend on the PCB. peppe iperf Done. Martin Blumenstingl (2): net: dt-bindings: add RGMII TX delay configuration to meson8b-dwmac net: stmmac: dwmac-meson8b: make the RGMII TX delay configurable Documentation/devicetree/bindings/net/meson-dwmac.txt | 11 +++ drivers/net/ethernet/stmicro/stmmac/dwmac-meson8b.c | 16 +++- include/dt-bindings/net/dwmac-meson8b.h | 18 ++ 3 files changed, 40 insertions(+), 5 deletions(-) create mode 100644 include/dt-bindings/net/dwmac-meson8b.h
Re: [net-next PATCH v1 0/2] stmmac: dwmac-meson8b: configurable RGMII TX delay
On Thu, Nov 24, 2016 at 7:55 PM, Florian Fainelliwrote: > Le 24/11/2016 à 09:05, Martin Blumenstingl a écrit : >> On Thu, Nov 24, 2016 at 4:56 PM, Jerome Brunet wrote: >>> On Thu, 2016-11-24 at 15:34 +0100, Martin Blumenstingl wrote: Currently the dwmac-meson8b stmmac glue driver uses a hardcoded 1/4 cycle TX clock delay. This seems to work fine for many boards (for example Odroid-C2 or Amlogic's reference boards) but there are some others where TX traffic is simply broken. There are probably multiple reasons why it's working on some boards while it's broken on others: - some of Amlogic's reference boards are using a Micrel PHY - hardware circuit design - maybe more... This raises a question though: Which device is supposed to enable the TX delay when both MAC and PHY support it? And should we implement it for each PHY / MAC separately or should we think about a more generic solution (currently it's not possible to disable the TX delay generated by the RTL8211F PHY via devicetree when using phy-mode "rgmii")? >>> >>> Actually you can skip the part which activate the Tx-delay on the phy >>> by setting "phy-mode = "rgmii-id" instead of "rgmii" >>> >>> phy->interface will no longer be PHY_INTERFACE_MODE_RGMII >>> but PHY_INTERFACE_MODE_RGMII_ID. >> unfortunately this is not true for RTL8211F (I did my previous tests >> with the same expectation in mind)! >> the code seems to suggest that TX-delay is disabled whenever mode != >> PHY_INTERFACE_MODE_RGMII. >> BUT: on my device RTL8211F_TX_DELAY is set even before >> "phy_write(phydev, 0x11, reg);"! > > (Adding Sebastian (and Mans, and Andrew) since he raised the same > question a while ago. I think I now understand a bit better what > Sebastian was after a couple of weeks ago) > >> >> Based on what I found it seems that rgmii-id, rgmii-txid and >> rgmii-rxid are supposed to be handled by the PHY. > > Correct, the meaning of PHY_INTERFACE_MODE should be from the > perspective of the PHY device: > > - PHY_INTERFACE_MODE_RGMII_TXID means that the PHY is responsible for > adding a delay when the MAC transmits (TX MAC -> PHY (delay) -> wire) > - PHY_INTERFACE_MODE_RGMII_RXID means that the PHY is responsible for > adding a delay when the MAC receives (RX MAC <- (delay) PHY) <- wire) and PHY_INTERFACE_MODE_RGMII_ID is basically _TXID and _RXID combined (meaning that the PHY is responsible for the TX and RX delays) >> That would mean that we have two problems here: >> 1) drivers/net/phy/realtek.c:rtl8211f_config_init should check for >> PHY_INTERFACE_MODE_RGMII_ID or PHY_INTERFACE_MODE_RGMII_TXID and >> enable the TX-delay in that case - otherwise explicitly disable it > > Agreed. (on a side-not: it seems that the RTL8211F's TX-delay setting is either untouched by a hardware reset via GPIO or enabled automatically during hardware reset via GPIO) >> 2) dwmac-meson8b.c should only use the configured TX-delay for >> PHY_INTERFACE_MODE_RGMII >> @Florian: could you please share your thoughts on this (who handles >> the TX delay in which case)? > > This also seems reasonable to do, provided that the PHY is also properly > configured not to add delays in both directions, and therefore assumes > that the MAC does it. on Amlogic Meson systems (at least on the ARM64 ones) all customer devices with Gbit ethernet are using the RTL8211F PHY. The only exception are some development/reference boards from Amlogic themselves, which seem to be using a Micrel RGMII PHY. > We have a fairly large problem with how RGMII delays are done in PHYLIB > and Ethernet MAC drivers (or just in general), where we can't really > intersect properly what a PHY is supporting (in terms of internal > delays), and what the MAC supports either. One possible approach could > be to update PHY drivers a list of PHY_INTERFACE_MODE_* that they > support (ideally, even with normalized nanosecond delay values), and > then intersect that with the requested phy_interface_t during > phy_{attach,connect} time, and feed this back to the MAC with a special > error code/callback, so we could gracefully try to choose another > PHY_INTERFACE_MODE_* value that the MAC supports > > A larger problem is that a number of drivers have been deployed, and > Device Trees, possibly with the meaning of "phy-mode" and > "phy-connection-type" being from the MAC perspective, and not the PHY > perspective *sigh*, good luck auditing those. > > So from there, here is possibly what we could do > > - submit a series of patches that update the PHYLIB documentation (there > are other things missing here) and make it clear from which entity (PHY > or MAC) does the delay apply to, document the "intersection" problem here sounds like a good idea, maybe we should move this to a separate thread (I guess this is the part which is especially interesting for Sebastian, Mans and Andrew)? > - have you document the
Re: [net-next PATCH v1 0/2] stmmac: dwmac-meson8b: configurable RGMII TX delay
Le 24/11/2016 à 09:05, Martin Blumenstingl a écrit : > On Thu, Nov 24, 2016 at 4:56 PM, Jerome Brunetwrote: >> On Thu, 2016-11-24 at 15:34 +0100, Martin Blumenstingl wrote: >>> Currently the dwmac-meson8b stmmac glue driver uses a hardcoded 1/4 >>> cycle TX clock delay. This seems to work fine for many boards (for >>> example Odroid-C2 or Amlogic's reference boards) but there are some >>> others where TX traffic is simply broken. >>> There are probably multiple reasons why it's working on some boards >>> while it's broken on others: >>> - some of Amlogic's reference boards are using a Micrel PHY >>> - hardware circuit design >>> - maybe more... >>> >>> This raises a question though: >>> Which device is supposed to enable the TX delay when both MAC and PHY >>> support it? And should we implement it for each PHY / MAC separately >>> or should we think about a more generic solution (currently it's not >>> possible to disable the TX delay generated by the RTL8211F PHY via >>> devicetree when using phy-mode "rgmii")? >> >> Actually you can skip the part which activate the Tx-delay on the phy >> by setting "phy-mode = "rgmii-id" instead of "rgmii" >> >> phy->interface will no longer be PHY_INTERFACE_MODE_RGMII >> but PHY_INTERFACE_MODE_RGMII_ID. > unfortunately this is not true for RTL8211F (I did my previous tests > with the same expectation in mind)! > the code seems to suggest that TX-delay is disabled whenever mode != > PHY_INTERFACE_MODE_RGMII. > BUT: on my device RTL8211F_TX_DELAY is set even before > "phy_write(phydev, 0x11, reg);"! (Adding Sebastian (and Mans, and Andrew) since he raised the same question a while ago. I think I now understand a bit better what Sebastian was after a couple of weeks ago) > > Based on what I found it seems that rgmii-id, rgmii-txid and > rgmii-rxid are supposed to be handled by the PHY. Correct, the meaning of PHY_INTERFACE_MODE should be from the perspective of the PHY device: - PHY_INTERFACE_MODE_RGMII_TXID means that the PHY is responsible for adding a delay when the MAC transmits (TX MAC -> PHY (delay) -> wire) - PHY_INTERFACE_MODE_RGMII_RXID means that the PHY is responsible for adding a delay when the MAC receives (RX MAC <- (delay) PHY) <- wire) > That would mean that we have two problems here: > 1) drivers/net/phy/realtek.c:rtl8211f_config_init should check for > PHY_INTERFACE_MODE_RGMII_ID or PHY_INTERFACE_MODE_RGMII_TXID and > enable the TX-delay in that case - otherwise explicitly disable it Agreed. > 2) dwmac-meson8b.c should only use the configured TX-delay for > PHY_INTERFACE_MODE_RGMII > @Florian: could you please share your thoughts on this (who handles > the TX delay in which case)? This also seems reasonable to do, provided that the PHY is also properly configured not to add delays in both directions, and therefore assumes that the MAC does it. We have a fairly large problem with how RGMII delays are done in PHYLIB and Ethernet MAC drivers (or just in general), where we can't really intersect properly what a PHY is supporting (in terms of internal delays), and what the MAC supports either. One possible approach could be to update PHY drivers a list of PHY_INTERFACE_MODE_* that they support (ideally, even with normalized nanosecond delay values), and then intersect that with the requested phy_interface_t during phy_{attach,connect} time, and feed this back to the MAC with a special error code/callback, so we could gracefully try to choose another PHY_INTERFACE_MODE_* value that the MAC supports A larger problem is that a number of drivers have been deployed, and Device Trees, possibly with the meaning of "phy-mode" and "phy-connection-type" being from the MAC perspective, and not the PHY perspective *sigh*, good luck auditing those. So from there, here is possibly what we could do - submit a series of patches that update the PHYLIB documentation (there are other things missing here) and make it clear from which entity (PHY or MAC) does the delay apply to, document the "intersection" problem here - have you document the configured behavior for dwmac-meson8b that we just discussed here in v2 of this patch series Sorry for the long post, here is a virtual potato: 0 -- Florian
Re: [net-next PATCH v1 0/2] stmmac: dwmac-meson8b: configurable RGMII TX delay
On Thu, Nov 24, 2016 at 4:56 PM, Jerome Brunetwrote: > On Thu, 2016-11-24 at 15:34 +0100, Martin Blumenstingl wrote: >> Currently the dwmac-meson8b stmmac glue driver uses a hardcoded 1/4 >> cycle TX clock delay. This seems to work fine for many boards (for >> example Odroid-C2 or Amlogic's reference boards) but there are some >> others where TX traffic is simply broken. >> There are probably multiple reasons why it's working on some boards >> while it's broken on others: >> - some of Amlogic's reference boards are using a Micrel PHY >> - hardware circuit design >> - maybe more... >> >> This raises a question though: >> Which device is supposed to enable the TX delay when both MAC and PHY >> support it? And should we implement it for each PHY / MAC separately >> or should we think about a more generic solution (currently it's not >> possible to disable the TX delay generated by the RTL8211F PHY via >> devicetree when using phy-mode "rgmii")? > > Actually you can skip the part which activate the Tx-delay on the phy > by setting "phy-mode = "rgmii-id" instead of "rgmii" > > phy->interface will no longer be PHY_INTERFACE_MODE_RGMII > but PHY_INTERFACE_MODE_RGMII_ID. unfortunately this is not true for RTL8211F (I did my previous tests with the same expectation in mind)! the code seems to suggest that TX-delay is disabled whenever mode != PHY_INTERFACE_MODE_RGMII. BUT: on my device RTL8211F_TX_DELAY is set even before "phy_write(phydev, 0x11, reg);"! Based on what I found it seems that rgmii-id, rgmii-txid and rgmii-rxid are supposed to be handled by the PHY. That would mean that we have two problems here: 1) drivers/net/phy/realtek.c:rtl8211f_config_init should check for PHY_INTERFACE_MODE_RGMII_ID or PHY_INTERFACE_MODE_RGMII_TXID and enable the TX-delay in that case - otherwise explicitly disable it 2) dwmac-meson8b.c should only use the configured TX-delay for PHY_INTERFACE_MODE_RGMII @Florian: could you please share your thoughts on this (who handles the TX delay in which case)? Regards, Martin
Re: [net-next PATCH v1 0/2] stmmac: dwmac-meson8b: configurable RGMII TX delay
On Thu, 2016-11-24 at 15:34 +0100, Martin Blumenstingl wrote: > Currently the dwmac-meson8b stmmac glue driver uses a hardcoded 1/4 > cycle TX clock delay. This seems to work fine for many boards (for > example Odroid-C2 or Amlogic's reference boards) but there are some > others where TX traffic is simply broken. > There are probably multiple reasons why it's working on some boards > while it's broken on others: > - some of Amlogic's reference boards are using a Micrel PHY > - hardware circuit design > - maybe more... > > This raises a question though: > Which device is supposed to enable the TX delay when both MAC and PHY > support it? And should we implement it for each PHY / MAC separately > or should we think about a more generic solution (currently it's not > possible to disable the TX delay generated by the RTL8211F PHY via > devicetree when using phy-mode "rgmii")? > > iperf3 results on my Mecool BB2 board (Meson GXM, RTL8211F PHY) with > TX clock delay disabled on the MAC (as it's enabled in the PHY > driver). > TX throughput was virtually zero before: > $ iperf3 -c 192.168.1.100 -R > Connecting to host 192.168.1.100, port 5201 > Reverse mode, remote host 192.168.1.100 is sending > [ 4] local 192.168.1.206 port 52828 connected to 192.168.1.100 port > 5201 > [ ID] Interval Transfer Bandwidth > [ 4] 0.00-1.00 sec 108 MBytes 901 > Mbits/sec > [ 4] 1.00-2.00 sec 94.2 MBytes 791 > Mbits/sec > [ 4] 2.00-3.00 sec 96.5 MBytes 810 > Mbits/sec > [ 4] 3.00-4.00 sec 96.2 MBytes 808 > Mbits/sec > [ 4] 4.00-5.00 sec 96.6 MBytes 810 > Mbits/sec > [ 4] 5.00-6.00 sec 96.5 MBytes 810 > Mbits/sec > [ 4] 6.00-7.00 sec 96.6 MBytes 810 > Mbits/sec > [ 4] 7.00-8.00 sec 96.5 MBytes 809 > Mbits/sec > [ 4] 8.00-9.00 sec 105 MBytes 884 > Mbits/sec > [ 4] 9.00-10.00 sec 111 MBytes 934 > Mbits/sec > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bandwidth Retr > [ 4] 0.00-10.00 sec 1000 MBytes 839 > Mbits/sec0 sender > [ 4] 0.00-10.00 sec 998 MBytes 837 > Mbits/sec receiver > > iperf Done. > $ iperf3 -c 192.168.1.100 > Connecting to host 192.168.1.100, port 5201 > [ 4] local 192.168.1.206 port 52832 connected to 192.168.1.100 port > 5201 > [ ID] Interval Transfer Bandwidth Retr Cwnd > [ 4] 0.00-1.01 sec 99.5 MBytes 829 Mbits/sec 117139 > KBytes > [ 4] 1.01-2.00 sec 105 MBytes 884 Mbits/sec 129 70.7 > KBytes > [ 4] 2.00-3.01 sec 107 MBytes 889 Mbits/sec 106187 > KBytes > [ 4] 3.01-4.01 sec 105 MBytes 878 Mbits/sec 92143 > KBytes > [ 4] 4.01-5.00 sec 105 MBytes 882 Mbits/sec 140129 > KBytes > [ 4] 5.00-6.01 sec 106 MBytes 883 Mbits/sec 115195 > KBytes > [ 4] 6.01-7.00 sec 102 MBytes 863 Mbits/sec 133 70.7 > KBytes > [ 4] 7.00-8.01 sec 106 MBytes 884 Mbits/sec 143 97.6 > KBytes > [ 4] 8.01-9.01 sec 104 MBytes 875 Mbits/sec 124107 > KBytes > [ 4] 9.01-10.01 sec 105 MBytes 876 Mbits/sec 90139 > KBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bandwidth Retr > [ 4] 0.00-10.01 sec 1.02 GBytes 874 > Mbits/sec 1189 sender > [ 4] 0.00-10.01 sec 1.02 GBytes 873 > Mbits/sec receiver > Cool, one more board working ;) I tried your patch on another board (MXQ_V2.1, with sheep printed on the PCB). It 's not improving the situation for this one unfortunately. Actually I already tried playing with the TX delay on the MAC and PHY but I could get any good results with the boards I have. It is strange that we can adjust the delay by 2ns steps, when delay seens by the phy should be 2ns ... > iperf Done. > > > Martin Blumenstingl (2): > net: dt-bindings: add RGMII TX delay configuration to meson8b-dwmac > net: stmmac: dwmac-meson8b: make the RGMII TX delay configurable > > Documentation/devicetree/bindings/net/meson-dwmac.txt | 11 > +++ > drivers/net/ethernet/stmicro/stmmac/dwmac-meson8b.c | 16 > +++- > include/dt-bindings/net/dwmac-meson8b.h | 18 > ++ > 3 files changed, 40 insertions(+), 5 deletions(-) > create mode 100644 include/dt-bindings/net/dwmac-meson8b.h >
Re: [net-next PATCH v1 0/2] stmmac: dwmac-meson8b: configurable RGMII TX delay
On Thu, 2016-11-24 at 15:34 +0100, Martin Blumenstingl wrote: > Currently the dwmac-meson8b stmmac glue driver uses a hardcoded 1/4 > cycle TX clock delay. This seems to work fine for many boards (for > example Odroid-C2 or Amlogic's reference boards) but there are some > others where TX traffic is simply broken. > There are probably multiple reasons why it's working on some boards > while it's broken on others: > - some of Amlogic's reference boards are using a Micrel PHY > - hardware circuit design > - maybe more... > > This raises a question though: > Which device is supposed to enable the TX delay when both MAC and PHY > support it? And should we implement it for each PHY / MAC separately > or should we think about a more generic solution (currently it's not > possible to disable the TX delay generated by the RTL8211F PHY via > devicetree when using phy-mode "rgmii")? Actually you can skip the part which activate the Tx-delay on the phy by setting "phy-mode = "rgmii-id" instead of "rgmii" phy->interface will no longer be PHY_INTERFACE_MODE_RGMII but PHY_INTERFACE_MODE_RGMII_ID. > > iperf3 results on my Mecool BB2 board (Meson GXM, RTL8211F PHY) with > TX clock delay disabled on the MAC (as it's enabled in the PHY > driver). > TX throughput was virtually zero before: > $ iperf3 -c 192.168.1.100 -R > Connecting to host 192.168.1.100, port 5201 > Reverse mode, remote host 192.168.1.100 is sending > [ 4] local 192.168.1.206 port 52828 connected to 192.168.1.100 port > 5201 > [ ID] Interval Transfer Bandwidth > [ 4] 0.00-1.00 sec 108 MBytes 901 > Mbits/sec > [ 4] 1.00-2.00 sec 94.2 MBytes 791 > Mbits/sec > [ 4] 2.00-3.00 sec 96.5 MBytes 810 > Mbits/sec > [ 4] 3.00-4.00 sec 96.2 MBytes 808 > Mbits/sec > [ 4] 4.00-5.00 sec 96.6 MBytes 810 > Mbits/sec > [ 4] 5.00-6.00 sec 96.5 MBytes 810 > Mbits/sec > [ 4] 6.00-7.00 sec 96.6 MBytes 810 > Mbits/sec > [ 4] 7.00-8.00 sec 96.5 MBytes 809 > Mbits/sec > [ 4] 8.00-9.00 sec 105 MBytes 884 > Mbits/sec > [ 4] 9.00-10.00 sec 111 MBytes 934 > Mbits/sec > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bandwidth Retr > [ 4] 0.00-10.00 sec 1000 MBytes 839 > Mbits/sec0 sender > [ 4] 0.00-10.00 sec 998 MBytes 837 > Mbits/sec receiver > > iperf Done. > $ iperf3 -c 192.168.1.100 > Connecting to host 192.168.1.100, port 5201 > [ 4] local 192.168.1.206 port 52832 connected to 192.168.1.100 port > 5201 > [ ID] Interval Transfer Bandwidth Retr Cwnd > [ 4] 0.00-1.01 sec 99.5 MBytes 829 Mbits/sec 117139 > KBytes > [ 4] 1.01-2.00 sec 105 MBytes 884 Mbits/sec 129 70.7 > KBytes > [ 4] 2.00-3.01 sec 107 MBytes 889 Mbits/sec 106187 > KBytes > [ 4] 3.01-4.01 sec 105 MBytes 878 Mbits/sec 92143 > KBytes > [ 4] 4.01-5.00 sec 105 MBytes 882 Mbits/sec 140129 > KBytes > [ 4] 5.00-6.01 sec 106 MBytes 883 Mbits/sec 115195 > KBytes > [ 4] 6.01-7.00 sec 102 MBytes 863 Mbits/sec 133 70.7 > KBytes > [ 4] 7.00-8.01 sec 106 MBytes 884 Mbits/sec 143 97.6 > KBytes > [ 4] 8.01-9.01 sec 104 MBytes 875 Mbits/sec 124107 > KBytes > [ 4] 9.01-10.01 sec 105 MBytes 876 Mbits/sec 90139 > KBytes > - - - - - - - - - - - - - - - - - - - - - - - - - > [ ID] Interval Transfer Bandwidth Retr > [ 4] 0.00-10.01 sec 1.02 GBytes 874 > Mbits/sec 1189 sender > [ 4] 0.00-10.01 sec 1.02 GBytes 873 > Mbits/sec receiver > > iperf Done. > > > Martin Blumenstingl (2): > net: dt-bindings: add RGMII TX delay configuration to meson8b-dwmac > net: stmmac: dwmac-meson8b: make the RGMII TX delay configurable > > Documentation/devicetree/bindings/net/meson-dwmac.txt | 11 > +++ > drivers/net/ethernet/stmicro/stmmac/dwmac-meson8b.c | 16 > +++- > include/dt-bindings/net/dwmac-meson8b.h | 18 > ++ > 3 files changed, 40 insertions(+), 5 deletions(-) > create mode 100644 include/dt-bindings/net/dwmac-meson8b.h >
[net-next PATCH v1 0/2] stmmac: dwmac-meson8b: configurable RGMII TX delay
Currently the dwmac-meson8b stmmac glue driver uses a hardcoded 1/4 cycle TX clock delay. This seems to work fine for many boards (for example Odroid-C2 or Amlogic's reference boards) but there are some others where TX traffic is simply broken. There are probably multiple reasons why it's working on some boards while it's broken on others: - some of Amlogic's reference boards are using a Micrel PHY - hardware circuit design - maybe more... This raises a question though: Which device is supposed to enable the TX delay when both MAC and PHY support it? And should we implement it for each PHY / MAC separately or should we think about a more generic solution (currently it's not possible to disable the TX delay generated by the RTL8211F PHY via devicetree when using phy-mode "rgmii")? iperf3 results on my Mecool BB2 board (Meson GXM, RTL8211F PHY) with TX clock delay disabled on the MAC (as it's enabled in the PHY driver). TX throughput was virtually zero before: $ iperf3 -c 192.168.1.100 -R Connecting to host 192.168.1.100, port 5201 Reverse mode, remote host 192.168.1.100 is sending [ 4] local 192.168.1.206 port 52828 connected to 192.168.1.100 port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 108 MBytes 901 Mbits/sec [ 4] 1.00-2.00 sec 94.2 MBytes 791 Mbits/sec [ 4] 2.00-3.00 sec 96.5 MBytes 810 Mbits/sec [ 4] 3.00-4.00 sec 96.2 MBytes 808 Mbits/sec [ 4] 4.00-5.00 sec 96.6 MBytes 810 Mbits/sec [ 4] 5.00-6.00 sec 96.5 MBytes 810 Mbits/sec [ 4] 6.00-7.00 sec 96.6 MBytes 810 Mbits/sec [ 4] 7.00-8.00 sec 96.5 MBytes 809 Mbits/sec [ 4] 8.00-9.00 sec 105 MBytes 884 Mbits/sec [ 4] 9.00-10.00 sec 111 MBytes 934 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.00 sec 1000 MBytes 839 Mbits/sec0 sender [ 4] 0.00-10.00 sec 998 MBytes 837 Mbits/sec receiver iperf Done. $ iperf3 -c 192.168.1.100 Connecting to host 192.168.1.100, port 5201 [ 4] local 192.168.1.206 port 52832 connected to 192.168.1.100 port 5201 [ ID] Interval Transfer Bandwidth Retr Cwnd [ 4] 0.00-1.01 sec 99.5 MBytes 829 Mbits/sec 117139 KBytes [ 4] 1.01-2.00 sec 105 MBytes 884 Mbits/sec 129 70.7 KBytes [ 4] 2.00-3.01 sec 107 MBytes 889 Mbits/sec 106187 KBytes [ 4] 3.01-4.01 sec 105 MBytes 878 Mbits/sec 92143 KBytes [ 4] 4.01-5.00 sec 105 MBytes 882 Mbits/sec 140129 KBytes [ 4] 5.00-6.01 sec 106 MBytes 883 Mbits/sec 115195 KBytes [ 4] 6.01-7.00 sec 102 MBytes 863 Mbits/sec 133 70.7 KBytes [ 4] 7.00-8.01 sec 106 MBytes 884 Mbits/sec 143 97.6 KBytes [ 4] 8.01-9.01 sec 104 MBytes 875 Mbits/sec 124107 KBytes [ 4] 9.01-10.01 sec 105 MBytes 876 Mbits/sec 90139 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth Retr [ 4] 0.00-10.01 sec 1.02 GBytes 874 Mbits/sec 1189 sender [ 4] 0.00-10.01 sec 1.02 GBytes 873 Mbits/sec receiver iperf Done. Martin Blumenstingl (2): net: dt-bindings: add RGMII TX delay configuration to meson8b-dwmac net: stmmac: dwmac-meson8b: make the RGMII TX delay configurable Documentation/devicetree/bindings/net/meson-dwmac.txt | 11 +++ drivers/net/ethernet/stmicro/stmmac/dwmac-meson8b.c | 16 +++- include/dt-bindings/net/dwmac-meson8b.h | 18 ++ 3 files changed, 40 insertions(+), 5 deletions(-) create mode 100644 include/dt-bindings/net/dwmac-meson8b.h -- 2.10.2