Hi Greg
Thanks for the advice. We want to keep the flows for at least 3 months
(ideally 6). Most of our queries will run on the last week's data - will
the cache drives help or not really? 10k SAS drives will effectively
cost 4x as much as the 7.2k - is the speed improvement worth it? RAID 10
seems to make sense although is less efficient (uses more drives) than
the initial RAID 6 but seems to be the logical choice for pure
performance - in RAID 6 we had 16x 4TB 7.2k drives. For RAID10 we'll
need 28x 7.2k drives or 62x1.8TB 10k drives (ouch). Isn't the read
performance the same though for both RAID levels and the important one
in this scenario? Ideally I want to go with RAID 10 10k SAS drives but
may have to settle with just RAID 10... PS reducing the RAM gets me very
few extra drives.
Oh yes, the included RAID chip is the SAS3108
(http://www.avagotech.com/products/storage/raid-on-chip/sas-3108). A
CacheVault kit has been included for protection. This seems good enough
- what do you think?
Regards,
Roderick
On 2016/11/01 9:56 PM, Greg Williams wrote:
>
> Hi Roderick, While we don’t process that much, only 4 Gbps and 500k
> pps peak, I have it running on an old core 2 duo with 6G of RAM. Yes,
> it is an old desktop system (8 years old). Nfsen uses barely any
> resources. I think your server is fine with the exception of
> storage. How long do you want to keep the flows? Your cache is only
> used for those queries where you are frequently accessing the data.
> Therefore, you will not see any speed increase with the SSD caching
> when searching for random data. I would highly recommend either
> 10/15k SAS or better yet, SSDs. Also look at RAID 10 for speed. If
> you are doing RAID 6, might as well do RAID 10. If cost is an issue,
> I would decrease the RAM and increase your storage. Also with data
> that important, make sure you get a good RAID card
> http://www.avagotech.com/products/storage/raid-controllers/megaraid-sas-9361-16i.
>
> If you just use a simple RAID card without a battery and large cache,
> you may corrupt your data in the event of a power failure.
>
>
>
> Greg Williams, ME
> Director of Networks and Infrastructure
> Information Technology
> University of Colorado Colorado Springs
> 1420 Austin Bluffs Parkway, (EPC 136A)
> Colorado Springs, CO 80918
> www.uccs.edu <http://www.uccs.edu/>
>
>
>
> *From:*Roderick Mooi [mailto:roder...@sanren.ac.za]
> *Sent:* Tuesday, November 1, 2016 7:17 AM
> *To:* nfdump-disc...@lists.sourceforge.net;
> nfsen-discuss@lists.sourceforge.net
> *Cc:* joh...@sanren.ac.za; Ishmael Tsoaela <ishm...@sanren.ac.za>
> *Subject:* [Nfsen-discuss] Netflow collector hardware advice
>
>
>
> Good day
>
> We would like to procure a server for netflow data collection and
> processing (nfdump + nfsen) and are looking for some hardware advice
> if you could please assist?
>
> Requirement:
> Import flow data from 4 x 10 Gbps router interfaces
> - currently we have 3 active interfaces with 5-6 Gbps of peak traffic
> each and between 500k and 1mil pps each
> - Juniper MX480 and Cisco ASR 9010
> - unsampled (full)
> Execute queries in a reasonable amount of time (ideally no more than a
> few minutes but I know it depends on the complexity and time frame
> which will vary depending on our use case at the time)
>
> Spec so far:
> Supermicro SSG-6028R-E1CR24N chassis
> Intel CPU: 4C E5-2623V4 2.6G 10M 8GT QPI
> 128GB DDR4 RAM [expandable]
> 60TB storage (10x6TB SAS 12Gb/s) RAID6 [expandable]
> 2 x 200GB SSD caching drives with LSI Cachecade Pro 2.0
>
>
> Questions:
>
> CPU
> - which is better for nfdump+nfsen - more cores or more speed? Or pros
> and cons of each? i.e. would a 10C E5-2630V4 2.2G 25M 8GT QP be a
> better or worse choice?
>
> Memory
> - how much is needed? (I figure it depends on the queries, so let's
> say we're looking for any connections containing a certain destination
> IP address within a 1 month period)
>
> Storage
> - we'd like to keep at least 3 months data for 4x10Gbps interfaces
> unsampled. How much do we need?
> - 4TB or 6TB drives (Seagate Makara)
> - read cache only?
>
> Thanks very much,
>
> Roderick
>
> --
> Engineer
> South African National Research Network (SANReN)
>
------------------------------------------------------------------------------
Developer Access Program for Intel Xeon Phi Processors
Access to Intel Xeon Phi processor-based developer platforms.
With one year of Intel Parallel Studio XE.
Training and support from Colfax.
Order your platform today. http://sdm.link/xeonphi
_______________________________________________
Nfsen-discuss mailing list
Nfsen-discuss@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/nfsen-discuss
