Re: The tale of a single MAC

[Seth Mattinen]

On 1/1/11 7:33 PM, Graham Wooden wrote:
> 
> So ­ here is the interesting part... Both servers are HP Proliant DL380 G4s,
> and both of their NIC1 and NIC2 MACs addresses are exactly the same.  Not
> spoofd and the OS drivers are not mucking with them ... They¹re burned-in ­
> I triple checked them in their respective BIOS screen.  I acquired these two
> machines at different times and both were from the grey market.  The ³What
> the ...² is sitting fresh in my mind ...  How can this be?
> 
> In the last 15 years of being in IT, I have never encountered a ³burned-in²
> duplicated MACs across two physically different machines.  What are the
> odds, that HP would dup¹d them and that both would eventually end up at my
> shop?  Or maybe this type of thing isn¹t big of deal... ?
> 


None of the HP servers I have contain duplicate MAC addresses. (I just
looked through all the iLO2 cards to make sure I wasn't lying.) I'll
send you some details offlist.

~Seth

Re: The tale of a single MAC

[Ian Henderson]

On 02/01/2011, at 2:33 PM, Graham Wooden wrote:

> I encountered an interesting issue today and I found it so bizarre – so I
> thought I would share it.

Had a fun one with D-Link ADSL modems a few years ago. The MAC address used to 
source PPPoE frames from the ADSL interface was the same in a batch of modems. 
This wasn't a problem when using our wholesaler's ATM based DSLAMs, but when we 
moved users to our own Ethernet based systems, we ran into some fun. 

The DSLAMs were configured to rewrite the user MAC address into a constructed 
address including their DSLAM port and PVC details toward the BRAS. Even though 
this rewrite was occurring correctly, within a single DSLAM unit two users with 
the same real MAC address would have five minutes of connectivity each, 
alternating between the two ports.

Rgds,



- I.

Re: The tale of a single MAC

[Graham Wooden]

Two different suppliers - one was out of Wisconsin (I believe; it's been
some time), and the other of Phoenix for the most recent batch.

I have lots and lots of HP server gear - and never encountered such bizarre
issue.


On 1/1/11 9:59 PM, "Brielle Bruns"  wrote:

> On 1/1/11 8:33 PM, Graham Wooden wrote:
>> So ­ here is the interesting part... Both servers are HP Proliant DL380 G4s,
>> and both of their NIC1 and NIC2 MACs addresses are exactly the same.  Not
>> spoofd and the OS drivers are not mucking with them ... They¹re burned-in ­
>> I triple checked them in their respective BIOS screen.  I acquired these two
>> machines at different times and both were from the grey market.  The ³What
>> the ...² is sitting fresh in my mind ...  How can this be?
> 
> 
>  From the same grey market supplier?
> 
> I know HP has a disc they put out which updates all the firmware/bios in
> a specific server model, its not too far fetched that a vendor might
> have a modified version that also either purposely or accidentally
> changes the MAC address.  Off the top of my head, I'm not sure where the
> MAC is stored - maybe an eeprom or a portion of the bios flash.  Or, it
> could be botched flashing that blew away the portion of memory where
> that was stored and the system defaulted to a built in value.
> 
> Excellent example is, IIRC, the older sparc stuff, where the ethernet
> cards didn't have MAC addresses as part of the card, but were stored in
> non-volatile or battery backed memory.  Memory goes poof, and you'll
> have problems.  Some WRT54G/WAP54Gs suffer from the same problem when
> throwing third party firmware on there.

Re: The tale of a single MAC

[Graham Wooden]

No - these are Genuine HP Servers.  Both servers have the latest BIOSs and
firmware applied to the board as well as cards.

The search results that I have seen didn't apply to the actual bios, rather
to guest Oss mucking or teamming.

On 1/1/11 9:56 PM, "Dobbins, Roland"  wrote:

> 
> On Jan 2, 2011, at 10:33 AM, Graham Wooden wrote:
> 
>>  What are the odds, that HP would dup¹d them and that both would eventually
>> end up at my shop?
> 
> 
> There may be some setting you're overlooking or a bug which needs an update to
> fix, or you may simply have purchased HP ProLiant *cases*, rather than actual
> *servers*.
> 
> ;>
> 
> Note that search engine results for 'proliant dl380 duplicate mac' returns
> multiple links.
> 
> 
> Roland Dobbins  // 
> 
> Most software today is very much like an Egyptian pyramid, with millions
> of bricks piled on top of each other, with no structural integrity, but
> just done by brute force and thousands of slaves.
> 
>  -- Alan Kay
> 
> 

Re: The tale of a single MAC

[bmanning]

i have seen dups in 3com, dell, and hp kit over the years.
the best was moving mac addresses btwn 802,3 and 802.5 cards.

--bill


On Sun, Jan 02, 2011 at 03:03:24PM +1030, Mark Smith wrote:
> On Sat, 01 Jan 2011 20:59:16 -0700
> Brielle Bruns  wrote:
> 
> > On 1/1/11 8:33 PM, Graham Wooden wrote:
> > > So - here is the interesting part... Both servers are HP Proliant DL380 
> > > G4s,
> > > and both of their NIC1 and NIC2 MACs addresses are exactly the same.  Not
> > > spoofd and the OS drivers are not mucking with them ... They9re burned-in 
> > > -
> > > I triple checked them in their respective BIOS screen.  I acquired these 
> > > two
> > > machines at different times and both were from the grey market.  The 3What
> > > the ...2 is sitting fresh in my mind ...  How can this be?
> > 
> > 
> >  From the same grey market supplier?
> > 
> > I know HP has a disc they put out which updates all the firmware/bios in 
> > a specific server model, its not too far fetched that a vendor might 
> > have a modified version that also either purposely or accidentally 
> > changes the MAC address.  Off the top of my head, I'm not sure where the 
> > MAC is stored - maybe an eeprom or a portion of the bios flash.  Or, it 
> > could be botched flashing that blew away the portion of memory where 
> > that was stored and the system defaulted to a built in value.
> > 
> > Excellent example is, IIRC, the older sparc stuff, where the ethernet 
> > cards didn't have MAC addresses as part of the card, but were stored in 
> > non-volatile or battery backed memory.
> 
> This was actually the intended way to use "MAC" addresses, to used as
> host addresses rather than as individual interface addresses, according
> to the following paper -
> 
> "48-bit Absolute Internet and Ethernet Host Numbers"
> Yogan K. Dalal and Robert S. Printis, July 1981
> http://ethernethistory.typepad.com/papers/HostNumbers.pdf
> 
> That paper also discusses why 48 bits were chosen as the size, despite
> "Ethernet systems" being limited to 1024 hosts. 
> 
> I think things evolved into MAC per NIC because when add-in NICs
> were invented there wasn't any appropriate non-volatile storage on the
> host to store the address. 
> 
> Regards,
> mark.
> 

Re: The tale of a single MAC

[Mark Smith]

On Sat, 01 Jan 2011 20:59:16 -0700
Brielle Bruns  wrote:

> On 1/1/11 8:33 PM, Graham Wooden wrote:
> > So ­ here is the interesting part... Both servers are HP Proliant DL380 G4s,
> > and both of their NIC1 and NIC2 MACs addresses are exactly the same.  Not
> > spoofd and the OS drivers are not mucking with them ... They¹re burned-in ­
> > I triple checked them in their respective BIOS screen.  I acquired these two
> > machines at different times and both were from the grey market.  The ³What
> > the ...² is sitting fresh in my mind ...  How can this be?
> 
> 
>  From the same grey market supplier?
> 
> I know HP has a disc they put out which updates all the firmware/bios in 
> a specific server model, its not too far fetched that a vendor might 
> have a modified version that also either purposely or accidentally 
> changes the MAC address.  Off the top of my head, I'm not sure where the 
> MAC is stored - maybe an eeprom or a portion of the bios flash.  Or, it 
> could be botched flashing that blew away the portion of memory where 
> that was stored and the system defaulted to a built in value.
> 
> Excellent example is, IIRC, the older sparc stuff, where the ethernet 
> cards didn't have MAC addresses as part of the card, but were stored in 
> non-volatile or battery backed memory.

This was actually the intended way to use "MAC" addresses, to used as
host addresses rather than as individual interface addresses, according
to the following paper -

"48-bit Absolute Internet and Ethernet Host Numbers"
Yogan K. Dalal and Robert S. Printis, July 1981
http://ethernethistory.typepad.com/papers/HostNumbers.pdf

That paper also discusses why 48 bits were chosen as the size, despite
"Ethernet systems" being limited to 1024 hosts. 

I think things evolved into MAC per NIC because when add-in NICs
were invented there wasn't any appropriate non-volatile storage on the
host to store the address. 

Regards,
mark.

Re: The tale of a single MAC

[Adrian Chadd]

So along simlar lines, Ubiquiti sell routerstation pro boards with
sequential MAC addresses.

The trouble is they've allocated a single MAC for the first port - the
second ethernet port (also attached to the bridge) doesn't get a second
MAC.

So in a purchase of a few hundred boards, we had plenty that were sequential.
Since the FreeBSD driver allocated MAC+1 to the second NIC, this caused
duplcate MAC addresses and this caused hilarity to ensue.

The "fix" was to just get this company to apply for some MAC space and then
use -that- on the second NIC and the bridge interfaces.

Ah, vendors.. :-)



Adrian

On Sat, Jan 01, 2011, Graham Wooden wrote:
> Hi there,
> 
> I encountered an interesting issue today and I found it so bizarre ? so I
> thought I would share it.
> 
> I brought online a spare server to help offload some of the recent VMs that
> I have been deploying.  Around the same time this new machine (we?ll call it
> Server-B) came online, another machine which has been online for about a
> year now stopped responding to our monitoring (and we?ll name this
> Server-A). I logged into the switch and saw that the machine that stopped
> responding was in the same VLAN as this newly deployed, and then quickly
> noticed that Server-A?s MAC address was now on Server-B?s switch port.
> ?What the ...? was my initial response.
> 
> I went ahead and moved Server-B?s to another VLAN, updated the switchport,
> cleared the ARP, and Server-A came back to life.  Happy new year to me.
> 
> So ? here is the interesting part... Both servers are HP Proliant DL380 G4s,
> and both of their NIC1 and NIC2 MACs addresses are exactly the same.  Not
> spoofd and the OS drivers are not mucking with them ... They?re burned-in ?
> I triple checked them in their respective BIOS screen.  I acquired these two
> machines at different times and both were from the grey market.  The ?What
> the ...? is sitting fresh in my mind ...  How can this be?
> 
> In the last 15 years of being in IT, I have never encountered a ?burned-in?
> duplicated MACs across two physically different machines.  What are the
> odds, that HP would dup?d them and that both would eventually end up at my
> shop?  Or maybe this type of thing isn?t big of deal... ?
> 
> -graham
> 
> 
> 

-- 
- Xenion - http://www.xenion.com.au/ - VPS Hosting - Commercial Squid Support -
- $24/pm+GST entry-level VPSes w/ capped bandwidth charges available in WA -

Re: The tale of a single MAC

[Dobbins, Roland]

On Jan 2, 2011, at 10:33 AM, Graham Wooden wrote:

>  What are the odds, that HP would dup’d them and that both would eventually 
> end up at my shop?


There may be some setting you're overlooking or a bug which needs an update to 
fix, or you may simply have purchased HP ProLiant *cases*, rather than actual 
*servers*.

;>

Note that search engine results for 'proliant dl380 duplicate mac' returns 
multiple links.


Roland Dobbins  // 

Most software today is very much like an Egyptian pyramid, with millions
of bricks piled on top of each other, with no structural integrity, but
just done by brute force and thousands of slaves.

  -- Alan Kay

Re: The tale of a single MAC

[Brielle Bruns]

On 1/1/11 8:33 PM, Graham Wooden wrote:

So ­ here is the interesting part... Both servers are HP Proliant DL380 G4s,
and both of their NIC1 and NIC2 MACs addresses are exactly the same.  Not
spoofd and the OS drivers are not mucking with them ... They¹re burned-in ­
I triple checked them in their respective BIOS screen.  I acquired these two
machines at different times and both were from the grey market.  The ³What
the ...² is sitting fresh in my mind ...  How can this be?



From the same grey market supplier?

I know HP has a disc they put out which updates all the firmware/bios in 
a specific server model, its not too far fetched that a vendor might 
have a modified version that also either purposely or accidentally 
changes the MAC address.  Off the top of my head, I'm not sure where the 
MAC is stored - maybe an eeprom or a portion of the bios flash.  Or, it 
could be botched flashing that blew away the portion of memory where 
that was stored and the system defaulted to a built in value.


Excellent example is, IIRC, the older sparc stuff, where the ethernet 
cards didn't have MAC addresses as part of the card, but were stored in 
non-volatile or battery backed memory.  Memory goes poof, and you'll 
have problems.  Some WRT54G/WAP54Gs suffer from the same problem when 
throwing third party firmware on there.


--
Brielle Bruns
The Summit Open Source Development Group
http://www.sosdg.org/ http://www.ahbl.org

Re: The tale of a single MAC

[Raul Rodriguez]

Seen this on six-figure gateways.

-RR

On 1/1/11, Suresh Ramasubramanian  wrote:
> I've seen duplicate MAC addresses but only on no name made in china
> NICs installed on cheap (assembled from parts) PCs at a school
> computer lab.
>
> On Sun, Jan 2, 2011 at 9:03 AM, Graham Wooden  wrote:
>>
>> In the last 15 years of being in IT, I have never encountered a
>> ³burned-in²
>> duplicated MACs across two physically different machines.  What are the
>> odds, that HP would dup¹d them and that both would eventually end up at my
>> shop?  Or maybe this type of thing isn¹t big of deal... ?
>
>
>
> --
> Suresh Ramasubramanian (ops.li...@gmail.com)
>
>

-- 
Sent from my mobile device

Re: The tale of a single MAC

[Suresh Ramasubramanian]

I've seen duplicate MAC addresses but only on no name made in china
NICs installed on cheap (assembled from parts) PCs at a school
computer lab.

On Sun, Jan 2, 2011 at 9:03 AM, Graham Wooden  wrote:
>
> In the last 15 years of being in IT, I have never encountered a ³burned-in²
> duplicated MACs across two physically different machines.  What are the
> odds, that HP would dup¹d them and that both would eventually end up at my
> shop?  Or maybe this type of thing isn¹t big of deal... ?



-- 
Suresh Ramasubramanian (ops.li...@gmail.com)

The tale of a single MAC

[Graham Wooden]

Hi there,

I encountered an interesting issue today and I found it so bizarre ­ so I
thought I would share it.

I brought online a spare server to help offload some of the recent VMs that
I have been deploying.  Around the same time this new machine (we¹ll call it
Server-B) came online, another machine which has been online for about a
year now stopped responding to our monitoring (and we¹ll name this
Server-A). I logged into the switch and saw that the machine that stopped
responding was in the same VLAN as this newly deployed, and then quickly
noticed that Server-A¹s MAC address was now on Server-B¹s switch port.
³What the ...² was my initial response.

I went ahead and moved Server-B¹s to another VLAN, updated the switchport,
cleared the ARP, and Server-A came back to life.  Happy new year to me.

So ­ here is the interesting part... Both servers are HP Proliant DL380 G4s,
and both of their NIC1 and NIC2 MACs addresses are exactly the same.  Not
spoofd and the OS drivers are not mucking with them ... They¹re burned-in ­
I triple checked them in their respective BIOS screen.  I acquired these two
machines at different times and both were from the grey market.  The ³What
the ...² is sitting fresh in my mind ...  How can this be?

In the last 15 years of being in IT, I have never encountered a ³burned-in²
duplicated MACs across two physically different machines.  What are the
odds, that HP would dup¹d them and that both would eventually end up at my
shop?  Or maybe this type of thing isn¹t big of deal... ?

-graham

Unusual BGP nexthop

[Ahsan Tariq]

Hi folks.

I am working on a project for which I am using the bgp routing tables from
routeviews and their update traces. In some update traces from routviews I
noticed that the nexthop field is set to: 255.255.255.255
Is this a valid nexthop ? Is their any meaning or significance to this
nexthop ?

Any help would be appreciated.

Thanks

Ahsan Tariq

Re: What sflow software - Manage Engine Net flow analyzer or Plixer Scrutinizer with Analyzer

[Peter Phaal]

sFlowTrend is free for up to five routers and should meet your requirement to 
quickly see top flows:

http://inmon.com/products/sFlowTrend.php

sFlowTrend is InMon's entry level product, if you need more features you might 
want to try sFlowTrend-Pro or Traffic Sentinel.

When selecting an sFlow analyzer, it is important to understand the sFlow 
architecture and the functional requirements it places on the analyzer - many 
products are principally netflow analyzers and do a poor job with sFlow

http://blog.sflow.com/2009/05/choosing-sflow-analyzer.html

Peter

On Jan 1, 2011, at 2:56 AM, Alex Pinto  wrote:

> 
> Hi everyone, we currently are looking at sflow options for a commercial 
> collector and analyzer. The core use is for visibility on our network, for 
> quickly detecting source / destination IP addresses, ie where the traffic is 
> going and where is it coming from, the type of traffic would be interesting 
> also but to be honest all which really matters is source / destination.
> 
> The requirement of the sflow software is to give us options and data very 
> quickly in the event of a DDOS attack so mitigation can occur quickly once we 
> understand what’s happening on the network. The last thing we want is for the 
> software not to work under a DDOS (too much data) thus leaving us blind upon 
> an attack. The quicker the software can report on issues, the quicker we can 
> do something about it. 
> Our current routers are fully sflow capable and both export nicely to both 
> packages.
> 
> Our findings so far
> 
> Manage Engine Net flow analyzer has both a Linux and windows version, the 
> software is very light and seems to perform very fast, although light on 
> additional features such as custom reporting, and alerting / in depth packet 
> information.  The concern is this software too simple, will it work under 
> heavy load?
> Based on our needs Manage Engine Net flow costs $2000.00
> 
> Plixer Scrutinizer – based on windows the software seems resource intensive 
> but has a MASSIVE amount of extra visibility built into the software 
> including automatic alerts, that being said the software does seem extremely 
> more complex to configure and understand, reports seem to take longer to 
> produce and the information doesn’t seem to be reported as quickly. (ie lags 
> by minutes or so compared to Manage Engine)  
> Based on our needs Plixer Scrutinizer Costs $4000.00
> 
> Does anyone have any real life experience on either package the cost 
> different between the two packages doesn’t worry us, it’s all about selecting 
> the correct package knowing the one time we need to access the flow 
> information and get it quick that the package we choose preforms quickly and 
> works.
> 
> I’d also like to hear from anyone else using another commercial solution, 
> which they would recommend.
> 
> Thanks in advance
> 
> Alex 

Re: What sflow software - Manage Engine Net flow analyzer or Plixer Scrutinizer with Analyzer

[Jake Wilson]

Alex Pinto  hotmail.com> writes:

> 
> 
> Hi everyone, we currently are looking at sflow options for a commercial 
collector and analyzer. The core
> use is for visibility on our network, for quickly detecting source / 
destination IP addresses, ie where
> the traffic is going and where is it coming from, the type of traffic would 
be interesting also but to be
> honest all which really matters is source / destination.
> 
> The requirement of the sflow software is to give us options and data very 
quickly in the event of a DDOS attack
> so mitigation can occur quickly once we understand what’s happening on the 
network. The last thing we
> want is for the software not to work under a DDOS (too much data) thus 
leaving us blind upon an attack. The
> quicker the software can report on issues, the quicker we can do something 
about it. 
> Our current routers are fully sflow capable and both export nicely to both 
packages.
> 
> Our findings so far
> 
> Manage Engine Net flow analyzer has both a Linux and windows version, the 
software is very light and seems to
> perform very fast, although light on additional features such as custom 
reporting, and alerting / in
> depth packet information.  The concern is this software too simple, will it 
work under heavy load?
> Based on our needs Manage Engine Net flow costs $2000.00
> 
> Plixer Scrutinizer – based on windows the software seems resource intensive 
but has a MASSIVE amount of
> extra visibility built into the software including automatic alerts, that 
being said the software does
> seem extremely more complex to configure and understand, reports seem to 
take longer to produce and the
> information doesn’t seem to be reported as quickly. (ie lags by minutes or 
so compared to Manage Engine)  
> Based on our needs Plixer Scrutinizer Costs $4000.00
> 
> Does anyone have any real life experience on either package the cost 
different between the two packages
> doesn’t worry us, it’s all about selecting the correct package knowing the 
one time we need to access
> the flow information and get it quick that the package we choose preforms 
quickly and works.
> 
> I’d also like to hear from anyone else using another commercial solution, 
which they would recommend.
> 
> Thanks in advance
> 
> Alex
> 


Hi Alex,

Scrutinizer saves 100% of the raw NetFlow data just like Wireshark.  Most 
collectors only keep what is in the tuple (e.g. src/dest IP, port, interface, 
protocol, autonomous system and few other fields). This makes the interface 
faster. If you export MAC address or VLANs, Scrutinizer will allow you to 
filter (and report in the next version) on these fields. Filtering and 
reporting is very important in traffic analysis.  We feel that the ability to 
Include/Exclude on any combination of fields is a must. ManageEngine can't do 
this.  

Scrutinizer saves much more flow data in the roll ups (up to 100K) than 
ManageEngine (e.g. ~1K) therefore the tables are much larger and slower to 
query in Scrutinizer although more accurate especially over time. I'm 
surprised that reports lag by minutes.  Here are some things to check:
 * is the router/switch sending > 3000 flows/second?
 * is the scrutinizer server under powered?
 * is antivirus running (it shouldn't be scanning the scrutinizer directory)

Did you see page 2 of this pdf: 
http://www.plixer.com/files/scrutinizer_netflow_challenge.pdf 

Does this help? 

Jake Wilson
plixer.com 

What sflow software - Manage Engine Net flow analyzer or Plixer Scrutinizer with Analyzer

[Alex Pinto]

Hi everyone, we currently are looking at sflow options for a commercial 
collector and analyzer. The core use is for visibility on our network, for 
quickly detecting source / destination IP addresses, ie where the traffic is 
going and where is it coming from, the type of traffic would be interesting 
also but to be honest all which really matters is source / destination.
 
The requirement of the sflow software is to give us options and data very 
quickly in the event of a DDOS attack so mitigation can occur quickly once we 
understand what’s happening on the network. The last thing we want is for the 
software not to work under a DDOS (too much data) thus leaving us blind upon an 
attack. The quicker the software can report on issues, the quicker we can do 
something about it. 
Our current routers are fully sflow capable and both export nicely to both 
packages.
 
Our findings so far
 
Manage Engine Net flow analyzer has both a Linux and windows version, the 
software is very light and seems to perform very fast, although light on 
additional features such as custom reporting, and alerting / in depth packet 
information.  The concern is this software too simple, will it work under heavy 
load?
Based on our needs Manage Engine Net flow costs $2000.00
 
Plixer Scrutinizer – based on windows the software seems resource intensive but 
has a MASSIVE amount of extra visibility built into the software including 
automatic alerts, that being said the software does seem extremely more complex 
to configure and understand, reports seem to take longer to produce and the 
information doesn’t seem to be reported as quickly. (ie lags by minutes or so 
compared to Manage Engine)  
Based on our needs Plixer Scrutinizer Costs $4000.00
 
Does anyone have any real life experience on either package the cost different 
between the two packages doesn’t worry us, it’s all about selecting the correct 
package knowing the one time we need to access the flow information and get it 
quick that the package we choose preforms quickly and works.
 
I’d also like to hear from anyone else using another commercial solution, which 
they would recommend.
 
Thanks in advance
 
Alex