Re: HW question: i210 vs. BCM5461S over SGMII: no response from PHY to MDIO requests?
...
yes I've just noticed Russell's patch mentioning mvneta,
and found the phylink patches to mvneta in net-next (until then I'd
been reading the vanilla, where they haven't landed yet).
https://git.kernel.org/pub/scm/linux/kernel/git/davem/net-next.git/tre
e/drivers/net/ethernet/marvell/mvneta.c
phylink_create() looks clear enough, but I got stuck once I looked at
phylink_register_sfp(). Which IMO asks the device tree to enumerate
driver "sfp" or some such. I'm lost again. Thanks for hinting that
the existing style of the Intel driver and the Linux "device tree"
don't mix well or readily.
It's sad. Mr. King's code is so obvious and easy to follow (except
maybe for the indirections through the device tree).
As for Intel software and support... that's a bit of love and hate on
my part :-) The more I get involved, the more intensive the mix.
On 21 Mar 2018 at 13:58, Andrew Lunn wrote:
> > I'd love to use existing code of the phylib to talk to the SFP
> > PHY's, maybe extend the phylib a bit (with the phy's I have), rather
> > than cobble together something crude and private on my own, inside
> > the igb driver.
>
> So this is quite a big job, to do it cleanly. You probably need to
> retain the intel code for MDIO/PHY/I2C, but add an option to make use
> of the Linux common MDIO/PHY/I2C infrastructure. Then you need to
> extend PHYLINK with a non device tree way to configure it, and glue
> all the parts together.
>
> You can make it a bit easier by just throwing away all the intel
> MDIO/PHY/I2C code, replacing it will Linux common code. But i expect
> the Intel maintainers would then reject your changes. There is too
> high a chance of introducing regressions.
>
thanks for that explanation :-) You have saved me some time.
Frankly I see no chance of my code getting accepted to upstream - I'm
critical to my own capabilities here.
int phylink_callback(char* msg)
{
printk(msg);
}
"Hello there, there's an SGMII SFP plugged into the port that you
have registered earlier, on the mdio-i2c bus you have set up,
we have configured the PHY for you, all you need to do is tuck this
leash up your MAC and tell it to try a handshake."
It would be lovely to add SFP hot-swap support and PHY auto-config to
the Intel driver, and it would possibly result in quite a voluminous
crapectomy in the Intel driver, but in my position the effort
required is clearly over my head, and, as you correctly say, I'm not
in the position to push this back upstream :-)
I'll see how far I can go, "low hanging fruit /halfway there" style.
For instance, the "previous-generation" API starting from
mdiobus_alloc() looks "bounded" in terms of effort.
(Example in drivers/net/ethernet/broadcom/tg3.c)
If I understand this correctly, it should allow me to tap into
phylib, only I'd have to give up the SFP EEPROM understanding
and PHY hot-swap (available from the phylink with SFP support).
I'll probably become silent now, for a while.
Thanks for all your polite explanations Andrew :-)
Frank
Re: HW question: i210 vs. BCM5461S over SGMII: no response from PHY to MDIO requests?
> Looking at the i2c dumps, and some past dumps from the igb driver, > it's dawning on me on me that the igb driver, without much hacking, > would try to read the PHY ID from the DMI/DDM block - a case which > the drivers/net/phy/mdio-i2c.c specifically avoids :-) It avoids if for a very good reason. This driver exports a standard Linux MDIO bus. The core phylib code will then probe the bus, read the IDs, find the correct PHY driver and loads it. It is probably good that you spend some time looking at a driver other than igb. Picking one i know a little, say the Freescale FEC. It has functions to perform MDIO read and MDIO write. These are then exported as an MDIO bus to the linux common code. And there are a few calls to phy_connect(), phy_start(), phy_stop(). That is how you build a driver which uses the code in drivers/net/phy. mvneta shows how you can use phylink. Study these two far a while. Andrew
Re: HW question: i210 vs. BCM5461S over SGMII: no response from PHY to MDIO requests?
> I was also wondering if someone has written any kernel-space support > for the SFP's. Sure enough, I've found lots of code by Russell King > under drivers/net/phy. I started reading from sfp.c, went on to > sfp-bus.c, next the phylink stuff... Answers lots of my questions. > Clearly someone has "been there and done that" - I mean how to > interpret SFP EEPROM bits and act upon them in the PHY > initialization. That is all quite new code. It has not spread too far yet. The Marvell mvneta ethernet driver is using it, and the Clearfog is probably the first in kernel board to make use of it. A few of us are working on converting DSA over to using PHYLINK, since we have boards with Ethernet switches and SFP connected to switch ports. > There are notes in the phylib drivers that this is "platform" stuff > - a keyword which speaks to me of stuff hardwired onboard in > embedded motherboards (is Russell King the father of Linux on ARM > ?), rather than general-purpose PnP and addon boards. At the moment, PHYLIB pretty much relies on device tree to glue all the parts together. There is no support for not using device tree at the moment. It would need somebody to contribute that code. The other issue is that the igb driver, like most of the intel Ethernet drivers, ignore much of the Linux common MDIO/PHY and I2C infrastructure, and does it all themselves. This is probably because they share code with the Windoze driver. > I'd love to use existing code of the phylib to talk to the SFP > PHY's, maybe extend the phylib a bit (with the phy's I have), rather > than cobble together something crude and private on my own, inside > the igb driver. So this is quite a big job, to do it cleanly. You probably need to retain the intel code for MDIO/PHY/I2C, but add an option to make use of the Linux common MDIO/PHY/I2C infrastructure. Then you need to extend PHYLINK with a non device tree way to configure it, and glue all the parts together. You can make it a bit easier by just throwing away all the intel MDIO/PHY/I2C code, replacing it will Linux common code. But i expect the Intel maintainers would then reject your changes. There is too high a chance of introducing regressions. Andrew
Re: HW question: i210 vs. BCM5461S over SGMII: no response from PHY to MDIO requests?
On 21 Mar 2018 at 11:47, Andrew Lunn , net...@vger.ker wrote: > Another question is, how to write the driver's initialization > sequence, for it to correctly decide if the SFP is SERDES or SGMII or > what. I could just follow the config obtained from the i210 EEPROM. > Alternatively, or somehow combined to that, I could try checking if > something responds to me over i2c, and do a quick check for SPD > EEPROM. If I find one, do a check for "MII regs over i2c" - at all > i2c slave addresses between 0x41..0x59 EXCEPT 0x50/0x51. > If I find MII-i2c regs, it's clear that the transceiver contains a > PHY, and wants to run in SGMII mode - otherwise it's a SERDES thing. > If nothing is found in i2c mode, and inherited i210 config indicates > SGMII, try external MDIO... if that fails, revert to SERDES. > I was also wondering if someone has written any kernel-space support for the SFP's. Sure enough, I've found lots of code by Russell King under drivers/net/phy. I started reading from sfp.c, went on to sfp-bus.c, next the phylink stuff... Answers lots of my questions. Clearly someone has "been there and done that" - I mean how to interpret SFP EEPROM bits and act upon them in the PHY initialization. And I keep wondering where to start :-) If I grep phylink recursively in drivers/net/ethernet, I get no hits... Any chance of hooking this up to the igb driver in a clean way? There are notes in the phylib drivers that this is "platform" stuff - a keyword which speaks to me of stuff hardwired onboard in embedded motherboards (is Russell King the father of Linux on ARM ?), rather than general-purpose PnP and addon boards. I'd love to use existing code of the phylib to talk to the SFP PHY's, maybe extend the phylib a bit (with the phy's I have), rather than cobble together something crude and private on my own, inside the igb driver. If you have any pointers, let me know. Frank
Re: HW question: i210 vs. BCM5461S over SGMII: no response from PHY to MDIO requests?
Just another follow-up: With specs on SFP MSA, DDM/DMI and MII in hand, I have determined: 0x50 (a.k.a. 0xA0 in SFP MSA spec) = the module's SPD "EEPROM" 0x51 (a.k.a. 0xA2 in SFP MSA spec) = diagnostics (DMI/DDM) 0x56 = MII management access over i2C Using eeprog (reading each offset twice to get 16bit words, which works for both the SFP's), I've been able to get the MII PHY ID's of the chips inside both modules: SFP#1 has PHY ID == 002060C1 == BCM5461S SFP#2 has PHY ID == 01410c97 == Marvell MV88E111x series. I don't know the precise model of the Marvell chip. Could be the mythical MV88E1113. The SPD EEPROM contents are interesting too. The module ID == 0 is weird, I would expect 0x03 = SFP, but then again, the SFP MSA doesn't consider SGMII in the socket at all, it appears to only consider SERDES, am I right? So skipping the module ID makes some limited sense to me. I haven't calculated the checksums yet, but multiple fields in there are filled in according to the SFP MSA spec. Interestingly, SFP#1 has "Encoding" (byte 11 / 0x0b) set to 0x05 = "SONET scrambled". But has other fields indicating 100Base-FX. For 100Base-FX I would expect encoding "4b/5b". Which is what SFP#2 has encoded in its otherwise sparse EEPROM. SFP#1 has DMI/DDM implemented. Alarm thresholds seem cranked to the limit, but some measured values are returned. SFP#2 does not have DMI/DDM active, yet it decodes the i2c device 0x51 (a.k.a. 0xA2) and about three bytes are non-empty... but holding nonsensical values (some thresholds). Looking at the i2c dumps, and some past dumps from the igb driver, it's dawning on me on me that the igb driver, without much hacking, would try to read the PHY ID from the DMI/DDM block - a case which the drivers/net/phy/mdio-i2c.c specifically avoids :-) My current opinion about the matters is, that I don't really need a valid SPD EEPROM to initialize the PHY in the SFP. The question is, if I can make the i210 properly handshake with the PHY on the SGMII payload lane. Another question is, how to write the driver's initialization sequence, for it to correctly decide if the SFP is SERDES or SGMII or what. I could just follow the config obtained from the i210 EEPROM. Alternatively, or somehow combined to that, I could try checking if something responds to me over i2c, and do a quick check for SPD EEPROM. If I find one, do a check for "MII regs over i2c" - at all i2c slave addresses between 0x41..0x59 EXCEPT 0x50/0x51. If I find MII-i2c regs, it's clear that the transceiver contains a PHY, and wants to run in SGMII mode - otherwise it's a SERDES thing. If nothing is found in i2c mode, and inherited i210 config indicates SGMII, try external MDIO... if that fails, revert to SERDES. And then there's the MII PHY internal config, which can be quite proprietary... And if I write all that, there's noone to re-test this after me, as my setup is such a special case :-/ No end of fun. Unfortunately, some other work is interfering at this very moment... Frank
Re: HW question: i210 vs. BCM5461S over SGMII: no response from PHY to MDIO requests?
On 20 Mar 2018 at 13:09, Andrew Lunn wrote: > > i2cdetect has found three i2c slaves (identical layout in both SFP's) > > at addresses 0x50, 0x51 and 0x56. > > What are they? EEPROM, DDM and "MDIO over i2c" ? > > The SFP's likely lack a proper SFP MSA data structure. > > 0x50 and 0x51 are EEPROM like. See drivers/net/phy/sfp.c. The standard > at24 EEPROM driver can also read it. And so should the SFP code in the > igb driver! > Yes - "should" is the right way to put it. Looking around, I'm wondering how much general-purpose hardware gets sold with an Intel gigabit chip and SFP sockets. Note: sockets for actual SFP's, rather than SFP-size transceivers soldered onboard (and hardwired in the i210 config EEPROM). It took me quite a while to find such a board from DeLock. I mean to say that I doubt how much actual testing on humans this i2c code has enjoyed :-) but admittedly I'm being cheeky here, I haven't reviewed that part of the driver. Frank
Re: HW question: i210 vs. BCM5461S over SGMII: no response from PHY to MDIO requests?
> i2cdetect has found three i2c slaves (identical layout in both SFP's) > at addresses 0x50, 0x51 and 0x56. > What are they? EEPROM, DDM and "MDIO over i2c" ? > The SFP's likely lack a proper SFP MSA data structure. 0x50 and 0x51 are EEPROM like. See drivers/net/phy/sfp.c. The standard at24 EEPROM driver can also read it. And so should the SFP code in the igb driver! > Does this make sense as some sort of proprietary standard > (Cisco, if I should take the vendor's label at face value), > or is this likely some "3rd-party innovation thinko", not worth > implementing? See if drivers/net/phy/mdio-i2c.c is useful. I'm just guessing here. Andrew
Re: HW question: i210 vs. BCM5461S over SGMII: no response from PHY to MDIO requests?
I've taken a look inside the two SFP's. http://support.fccps.cz/download/adv/frr/ptp/inside_sfps.zip The uglier, bigger and likely older model (my SFP#2) contains two PCB's sandwiched, and the key chips are inside the sandwich. Thus, the photoes don't show much. The sexier SFP#1 = the one with the Broadcom chip... I believe it's what it says on the tin: - the BCM5461SA is directly interfaced to a laser driver chip (pretty much analog, found a datasheet from Maxim) - the RX direction seems equally simple, I haven't identified the chip by the marking, but by pinout it's undoubtedly an RX amplifier (they're called transimpedance amps, are they?) and, looking at the PCB traces, its output is directly interfaced to the BCM5461SA. - there's another BGA chip on the board, smaller than the BCM PHY. I believe that this is an "SFP MCU". Might be from Maxim or from someone completely different :-) Not sure whose trademark the "+" sign is in the chip marking. I've found a *slightly* more promising "data brief" from Broadcom for the BCM5461S: http://infiber.ru/assets/files/products/phy/BCM5461S_Product_Brief.pdf This one mentions 100Base-FX among the interface formats supported. I don't believe that the SFP maker would pipe copper MLT3 into the fiber driver+receiver :-) Note that the block diagram in the Broadcom PDF contains other minor errors. God knows what the true functionality is. The SFP#1 PCB still doesn't look fake to me. Frank
Re: HW question: i210 vs. BCM5461S over SGMII: no response from PHY to MDIO requests?
> I'm not getting an ACK from the SFP, probably because I've got the > address and offset wrong and because I'd better use indirect access. > There's some more work awaiting me... Try address 0x50. i2detect will probe all addresses for you, if you have a standard Linux i2c bus. Andrew
Re: HW question: i210 vs. BCM5461S over SGMII: no response from PHY to MDIO requests?
> > > Right now I've modded igb_init_i2c() to engage the bit-banging
> > > i2c driver for the i210 too
> >
> > I don't think that will work. The datasheet for the i210 talks about
> > two registers for I2C/MDIO which are not bit-banging. Only the i350
> > uses bit-banging.
> >
> From the i210 datasheet, page 477:
> chapter 8.17.9 "SFP I2C Parameters" - reg. I2CPARAMS (0x102C; R/W)
> bit 8 "I2CBB_EN" = I2C bit-bang enable.
> And about 6 more bits for SDA and SCL direction, input and output.
> Looking through existing code of the bit-banging callbacks for i350,
> their function would seem identical between the i210 and i350.
> I may check the bit definition macros again, if the scope shows
> nothing.
>
Sure enough, the I2C port works in bit-banging mode, even without a
semaphore. Screenshot attached.
I'm not getting an ACK from the SFP, probably because I've got the
address and offset wrong and because I'd better use indirect access.
There's some more work awaiting me...
God knows if this is going to be any use :-)
Frank
The following section of this message contains a file attachment
prepared for transmission using the Internet MIME message format.
If you are using Pegasus Mail, or any other MIME-compliant system,
you should be able to save it or view it from within your mailer.
If you cannot, please ask your system administrator for assistance.
File information ---
File: osc_some_frame_bitbang.png
Date: 17 Mar 2018, 17:43
Size: 54383 bytes.
Type: Unknown
osc_some_frame_bitbang.png
Description: Binary data
The following section of this message contains a file attachment
prepared for transmission using the Internet MIME message format.
If you are using Pegasus Mail, or any other MIME-compliant system,
you should be able to save it or view it from within your mailer.
If you cannot, please ask your system administrator for assistance.
File information ---
File: osc_some_frame_HW.png
Date: 17 Mar 2018, 17:39
Size: 40613 bytes.
Type: Unknown
osc_some_frame_HW.png
Description: Binary data
Re: HW question: i210 vs. BCM5461S over SGMII: no response from PHY to MDIO requests?
On 17 Mar 2018 at 15:50, Andrew Lunn wrote:
> On Sat, Mar 17, 2018 at 08:39:00AM +0100, Frantisek Rysanek wrote:
> > On 16 Mar 2018 at 22:02, Andrew Lunn wrote:
> > >
> > > Does ethtool -m show anything useful?
> > >
> >
> > Not much. "unsupported".
>
> static int igb_get_module_info(struct net_device *netdev,
>struct ethtool_modinfo *modinfo)
> {
> struct igb_adapter *adapter = netdev_priv(netdev);
> struct e1000_hw *hw = >hw;
> u32 status = 0;
> u16 sff8472_rev, addr_mode;
> bool page_swap = false;
>
> if ((hw->phy.media_type == e1000_media_type_copper) ||
> (hw->phy.media_type == e1000_media_type_unknown))
> return -EOPNOTSUPP;
>
> Suggests that the driver does not know you have a fibre module.
>
Good! I'll check this out.
Even if it means that I have to hack the "sequence of initialization"
in the flow of driver code... (again.)
> > Right now I've modded igb_init_i2c() to engage the bit-banging
> > i2c driver for the i210 too
>
> I don't think that will work. The datasheet for the i210 talks about
> two registers for I2C/MDIO which are not bit-banging. Only the i350
> uses bit-banging.
>
>From the i210 datasheet, page 477:
chapter 8.17.9 "SFP I2C Parameters" - reg. I2CPARAMS (0x102C; R/W)
bit 8 "I2CBB_EN" = I2C bit-bang enable.
And about 6 more bits for SDA and SCL direction, input and output.
Looking through existing code of the bit-banging callbacks for i350,
their function would seem identical between the i210 and i350.
I may check the bit definition macros again, if the scope shows
nothing.
If the HW implementation of the I2C protocol doesn't work for some
reason, I can as well try bit-banging it.
Thanks for your continued input :-)
Frank
Re: HW question: i210 vs. BCM5461S over SGMII: no response from PHY to MDIO requests?
On Sat, Mar 17, 2018 at 08:39:00AM +0100, Frantisek Rysanek wrote:
> On 16 Mar 2018 at 22:02, Andrew Lunn wrote:
> >
> > Does ethtool -m show anything useful?
> >
>
> Not much. "unsupported".
static int igb_get_module_info(struct net_device *netdev,
struct ethtool_modinfo *modinfo)
{
struct igb_adapter *adapter = netdev_priv(netdev);
struct e1000_hw *hw = >hw;
u32 status = 0;
u16 sff8472_rev, addr_mode;
bool page_swap = false;
if ((hw->phy.media_type == e1000_media_type_copper) ||
(hw->phy.media_type == e1000_media_type_unknown))
return -EOPNOTSUPP;
Suggests that the driver does not know you have a fibre module.
> Right now I've modded igb_init_i2c() to engage the bit-banging
> i2c driver for the i210 too
I don't think that will work. The datasheet for the i210 talks about
two registers for I2C/MDIO which are not bit-banging. Only the i350
uses bit-banging.
Andrew
Re: HW question: i210 vs. BCM5461S over SGMII: no response from PHY to MDIO requests?
On 16 Mar 2018 at 22:02, Andrew Lunn wrote: > > Does ethtool -m show anything useful? > Not much. "unsupported". Probably the ioctl() is not implemented or something, I haven't investigated. Maybe I should. Right now I've modded igb_init_i2c() to engage the bit-banging i2c driver for the i210 too, if SGMII active and external MDIO off. Which gets me a /dev/i2c-8 created (with modprobe i2c-dev), and I'm trying an 8bit read transaction on slave addresses from 0 to 127, which yields no responses. Maybe the 8bit transaction is not the right way to do it, maybe I should use indirect access, I'm just a little scared of that, I'd hate to reprogram the SFP inadvertently :-) I know that I'd have to pipe this bit-banging stuff into the Ethtool ioctl handling stuff, to get it to work via ethtool -m. The way I have hacked the driver, I've noticed that if I enable the BB mode in I2CPARAMS, the driver won't load completely, because the HW-based I2C controller is defunct, and the driver gets nothing at all when trying to read the PHY_ID. But if I first leave I2CPARAMS configured for HW-driven I2C, load the driver, and then switch I2CPARAMS to BB mode, I get the bit-banged I2C device loaded and the bit-twiddling callbacks in igb.ko do get called. I'm currently using devmem2 to tweak I2CPARAMS: devmem2 0xF750102C w 0x7646 devmem2 0xF750102C w 0x7746 I should probably decide which way to go, HW i2c or SW i2c :-) and mod the driver accordingly. I'm currently looking at the bb i2c callbacks in igb.ko with some suspicion... they seem a little too coarse to me. I'd expect some finer timing around "tri-state the pin, wait a few us and only then read the input". E.g. when checking for an ACK from the slave. I'm not sure if I should "take the i210 HW semaphore" while bit-banging the i2c stuff. Currently the BB code doesn't do that. Not sure if this prevents the bits from getting in and out at the PCB traces. Will have to check on monday. I've also noticed the drivers/net/phy/mdio-i2c.c . Maybe I should engage this instead or on top of the generic I2C bit-banging framework :-) => I haven't tried all my ideas yet and will have to stop tweaking for now... I'm playing with this remotely at home via SSH, so I don't see the 'scope (if anything happens on the bus in the BB mode). And it's time for breakfast :-) Frank
Re: HW question: i210 vs. BCM5461S over SGMII: no response from PHY to MDIO requests?
> The DeLock board is this beauty: > http://www.delock.de/produkte/G_89481/merkmale.html?setLanguage=en > DeLock techsupport were so kind as to provide me with a schematic > snippet, showing the wiring of the i210 fiber SKU's dedicated > I2C/MDIO pins to the SFP socket's standard I2C pins. There are > pull-up resistors on the board for SDA/SCL. Nothing outright custom > here - more likely close to some Intel reference design. The board > works just fine. I would suggest you ignore MDIO and try looking on the I2C bus. Does ethtool -m show anything useful? Andrew
Re: HW question: i210 vs. BCM5461S over SGMII: no response from PHY to MDIO requests?
On 16 Mar 2018 at 19:42, Andrew Lunn wrote: > Hi Frantisek > > This seems a bit odd. The SFP cage only has i2c, not MDIO. It is > possible to carry MDIO over i2c, which is what is done when the SFP > module is copper, not fibre. But you are doing 100Base-FX, so fibre. > > The BCM5461 is a copper PHY. There are some PHYs which are designed to > act as translators. You can feed SERDES in one side, and get SGMII out > the other side, for example. Such PHYs can be used between a MAC and > an SFP socket. But from what i can find online about the BCM5461, it > does not have this capability. > > Can you tell us the exact make/model of the DeLock card and the SFP > modules. > > Andrew > Hello Andrew, thanks for your polite response. The DeLock board is this beauty: http://www.delock.de/produkte/G_89481/merkmale.html?setLanguage=en DeLock techsupport were so kind as to provide me with a schematic snippet, showing the wiring of the i210 fiber SKU's dedicated I2C/MDIO pins to the SFP socket's standard I2C pins. There are pull-up resistors on the board for SDA/SCL. Nothing outright custom here - more likely close to some Intel reference design. The board works just fine. Interesting - I've noticed before that the sparse Broadcom data brief for the BCM5461S doesn't cotain a word about 100Base-FX. This might be a good question to the SFP vendor's techsupport :-) The 54616S datasheet does mention 100Base-FX, but I have no reason to believe that my SFP contains that chip. It was mostly my naive assumption that the BCM5461S would speak MDIO/MDC via the SFP slot's I2C lines :-) The i210 pinout allows for that and the BCM5461S doesn't seem to have I2C for management. It does have MDIO and MDC. I've tried the "SGMII with i2c management" option too - as I said the SFP seems to be responding, but the response is some junk. Not corresponding to what the Intel driver expects. As for the SFP's... The one that I actually bought as shiny new and speaking SGMII, and that contains a chip labeled 5461SA, is sold as Huwei-compatible. I do not know what that means at pinout level and protocol level :-) "got it at e-bay" says it all. Yet, I have to say that I've been politely and firmly warned by the selling party that it's not gonna work in my Intel board. I bought it anyway, out of curiosity. So I'm not going to provide a link, unless I find something to praise on the product :-) I cannot exclude the possibility that the SFP is a fake, despite being mechanically/visually a masterpiece. The other SFP I got for free, with a comment that "it's some Chinese junk, supposedly SGMII compatible, but it hasn't ever worked for us in any hardware. Here you go, have it." And the chips inside are obscured between two PCB's = no idea what they are, unless I take a hot air pen to it. I'll probably take some macro photoes of the SFP internals (on monday) and post them. Perhaps the set of chips may ring a bell with someone here :-) For the moment, thanks for your attention. Frank Rysanek Customer support FCC prumyslove systemy s.r.o. Usti nad Labem Czech Republic +420 47 2774 173 (office) +420 736 630 449 (mobile) rysa...@fccps.cz
Re: HW question: i210 vs. BCM5461S over SGMII: no response from PHY to MDIO requests?
On Fri, Mar 16, 2018 at 05:48:20PM +0100, Frantisek Rysanek wrote: > Dear polite inhabitants of the "netdev" mailing list, > > this is for a skunkworks project at the fringe of my job... > More of a DIY hobby thing :-) I'm tinkering and having fun. > > The wizards from linux-ptp have taught me how to use the i210 for > precise timestamping, which works fine at all copper speeds > and likely also for gigabit fiber (SERDES) which the i210 natively > supports (and which links fine). > > The catch is: I'm trying to get this to work on 100Base-FX :-) > which is not a native interface for the i210. > It can do SERDES only at 1Gb. But, it can do SGMII. > I've managed to find an i210-based NIC from DeLock with an SFP slot > and I've managed to find two models of 100Base-FX SGMII SFP's. > > My assumption is that the SFP's have SGMII instead of SERDES > and MDIO instead of SPD I2C - and the manufacturers of the board > and of the SFP's don't dispute that. Yet they have both vaguely > warned me in advance that it would not work work :-) > Well I've given it a try anyway, and I'm stuck at the MDIO level. > > With one of the SFP's, I know for a fact that it's based on the > BCM5461S. The actual marking on the chip goes BCM5461SA... etc. Hi Frantisek This seems a bit odd. The SFP cage only has i2c, not MDIO. It is possible to carry MDIO over i2c, which is what is done when the SFP module is copper, not fibre. But you are doing 100Base-FX, so fibre. The BCM5461 is a copper PHY. There are some PHYs which are designed to act as translators. You can feed SERDES in one side, and get SGMII out the other side, for example. Such PHYs can be used between a MAC and an SFP socket. But from what i can find online about the BCM5461, it does not have this capability. Can you tell us the exact make/model of the DeLock card and the SFP modules. Andrew
HW question: i210 vs. BCM5461S over SGMII: no response from PHY to MDIO requests?
Dear polite inhabitants of the "netdev" mailing list, this is for a skunkworks project at the fringe of my job... More of a DIY hobby thing :-) I'm tinkering and having fun. The wizards from linux-ptp have taught me how to use the i210 for precise timestamping, which works fine at all copper speeds and likely also for gigabit fiber (SERDES) which the i210 natively supports (and which links fine). The catch is: I'm trying to get this to work on 100Base-FX :-) which is not a native interface for the i210. It can do SERDES only at 1Gb. But, it can do SGMII. I've managed to find an i210-based NIC from DeLock with an SFP slot and I've managed to find two models of 100Base-FX SGMII SFP's. My assumption is that the SFP's have SGMII instead of SERDES and MDIO instead of SPD I2C - and the manufacturers of the board and of the SFP's don't dispute that. Yet they have both vaguely warned me in advance that it would not work work :-) Well I've given it a try anyway, and I'm stuck at the MDIO level. With one of the SFP's, I know for a fact that it's based on the BCM5461S. The actual marking on the chip goes BCM5461SA... etc. I've managed to modify the EEPROM of the NIC with EEUPDATE. The Device ID wouldn't change, but the "flash config words" that I needed were all willing to accept the change. So I can switch the chip to SGMII with external MDIO, and I can even see the chip generate an MDIO read transaction right after power-up. http://support.fccps.cz/download/adv/frr/ptp/MDIO_oscillograms.zip And this is where I'm stuck: the BCM chip does not respond to the i210 MDIO requests. I've hacked the igb driver a little, adding printks where I needed them, to see what's going on... I believe I've also spotted a minor bug where some PHY detection routine tries to read the status register before the phy.addr is even initialized... This is what the Intel chip does on power up: "read PHY reg. 0 from PHY addr ". (= read PHY_CONTROL) The igb driver does: 1) ret_val = hw->phy.ops.write_reg(hw, 0x1B, 0x8084); ...which succeeds, as apparently there's no "ACK bit" in the write sequence. The MDIO master never gets to know if the slave PHY is alive or not :-) 2) read "PHY control" == read PHY register 0 ... this fails. The PHY just doesn't do its part, and the MDIO controller HW in the MAC doesn't raise the status flag saying "transaction completed". The PHY just doesn't take over. It seems as if the i210 does not even continue sending CLK pulses - as if waiting for an active 0 (ACK?) from the PHY, and the ACK never comes. I have modified the igb driver to just go ahead and try the next step: 3) read PHY device ID == read PHY_ID1 == PHY register 0x02. ...which fails. I did not bother to try other registers. The one thing I'm not certain about is the required PHY addr. Not sure if this is hardwired in the PHY chip, or set by pin straps, or set by soft straps in the SFP's EEPROM or something. (Not that I've seen an EEPROM inside the SFP.) I've noticed some earlier suggested patches by Jonathan Toppins and Alan Liebthal of Cumulus Networks (from mid 2015) who seem to have gotten the "server ATOM's" MAC to work with a BCM5461S. Their code mentioned phy_addr = 5, which was possibly board-specific in their hardware. I've actually tried wrapping the steps 1) 2) 3) above in a FOR loop, iterating over phy_addr from 1 to 31. I can see on an oscilloscope that the i210's MDIO controller keeps trying different PHY addresses in the respective 5 bits of phy_addr, but the SFP just doesn't respond to any read transaction. The MDC (clock) from the i210 is a nice 50:50 rectangle at 2 MHz. I have reasons to believe that the SFP is powered. There's a P-FET in 3.3Vcc driven by SDP3, which opens when SDP3 goes log.0. Which does happen - the required state of SDP3 is encoded in the flash and the driver also writes the CTRL_EXT at runtime just in case. Also, it works in SERDES mode, where the NIC links just fine. And, I believe I can see some erratic responses from the SFP if I turn off external MDIO = when the i210 SGMII tries to run with i2c for management access to the PHY. In i2c mode, the controller runs the clock line at 100 kHz (and obviously the framing/protocol is different) - seems to me on the scope that the PHY starts babbling something on the data line. The read transactions in i2c mode even seem to succeed, but they read garbage. Interestingly regular, garbage. Mii-diag says: Basic registers of MII PHY #1: 7fff ff80 8000 007f 7fff ff80 8000 00ff. And that pattern repeats always the same. The SPF vendor has already replied that, as per spec, the BCM PHY should support up to 2.5 MHz MDC. And, they provided a framing diagram which looks like a perfect IEEE clause 22. Still I'm wondering if I should try to marry the igb driver with the "phylib" (which has some specific support for the BCM5461) and specifically with the bit-banging MDIO driver. Bit-banging might get me around some
