David sorry for the delay. What you can also do is the following 1. Enable RSS let’s say with two queues 2. start nprobe -i eth1@0,eth2@0 -g 1 ... nprobe -i eth1@1,eth2@1 -g 2 ...
If this is not enough you can increase the number of RSS queues so that each probe has less messages to process Regards Luca > On 22 Jan 2018, at 17:41, David Notivol <[email protected]> wrote: > > Hello, > > Sorry for replying to myself. > > Just adding troubleshooting information. We've tested using zbalance to form > the virtual interface instead of our application, and we are getting the same > results; we keep having drops. > Thanks. > > Regards, > David Notivol. > > 2018-01-19 16:59 GMT+01:00 David Notivol <[email protected] > <mailto:[email protected]>>: > Hello list, > > I'm testing nProbe listening from two different 10Gb interfaces (using i40e > pf_ring's driver). As we need to mix up the information of these two links, > we use a custom application (using pf_ring sources) that creates a virtual > interface with traffic from the original physical ones. > > The total traffic is about 3Gbps (we expect to have more), but we are seeing > around 50-60% packet drops between our application and nProbe. When testing > with zcount/pfcount instead of nProbe, we see 0% drops. > > I've made some tuning in the nProbe parameters (hash-size, max-num-flows, > idle-timeout, ...), but no significant changes has been noticed. > > Drops are fewer when disabling the export to Kafka and disabling all plugins > (GTPv1, GTPv2 and HTTP), although we always have (around 1-2%). > > I'm pasting below my nProbe configuration, and some traffic statistics. > > Do you have any recommendation I could follow to improve this performance? > Thanks a lot in advance. > > > > -- System: > > nProbe: nprobe-8.0.171020-5797.x86_64 > System RAM: 64GB > System CPU: 12 cores > System OS: CentOS Linux release 7.4.1708 (Core) > Linux Kernel: 3.10.0-693.11.6.el7.x86_64 #1 SMP Thu Jan 4 01:06:37 UTC 2018 > x86_64 x86_64 x86_64 GNU/Linux > > > -- nProbe configuration: > -n=none > -i=zc:1@0 > -s=128 > -t=60 > -d=30 > -a=0 > -e=1 > -B=10 > -w=4048000 > -M=10000000 > -z=0 > -S=1:1 > -E=0:0 > -g=/var/run/nprobe-zc1-0.pid > --vlanid-as-iface-idx=none > -V=9 > -T="%IPV4_SRC_ADDR %IPV4_DST_ADDR %IN_PKTS %IN_BYTES %OUT_PKTS %OUT_BYTES > %FIRST_SWITCHED %LAST_SWITCHED %L4_SRC_PORT %L4_DST_PORT %TCP_FLAGS %PROTOCOL > %SRC_TOS %SRC_AS %DST_AS %L7_PROTO %L7_PROTO_NAME %SRC_IP_COUNTRY > %SRC_IP_CITY %SRC_IP_LONG %SRC_IP_LAT %DST_IP_COUNTRY %DST_IP_CITY > %DST_IP_LONG %DST_IP_LAT %SRC_VLAN %DST_VLAN %DOT1Q_SRC_VLAN %DOT1Q_DST_VLAN > %DIRECTION %SSL_SERVER_NAME %SRC_AS_MAP %DST_AS_MAP %HTTP_METHOD > %HTTP_RET_CODE %HTTP_REFERER %HTTP_UA %HTTP_MIME %HTTP_HOST %HTTP_SITE > %UPSTREAM_TUNNEL_ID %UPSTREAM_SESSION_ID %DOWNSTREAM_TUNNEL_ID > %DOWNSTREAM_SESSION_ID %UNTUNNELED_PROTOCOL %UNTUNNELED_IPV4_SRC_ADDR > %UNTUNNELED_L4_SRC_PORT %UNTUNNELED_IPV4_DST_ADDR %UNTUNNELED_L4_DST_PORT > %GTPV2_REQ_MSG_TYPE %GTPV2_RSP_MSG_TYPE %GTPV2_C2S_S1U_GTPU_TEID > %GTPV2_C2S_S1U_GTPU_IP %GTPV2_S2C_S1U_GTPU_TEID %GTPV2_S5_S8_GTPC_TEID > %GTPV2_S2C_S1U_GTPU_IP %GTPV2_C2S_S5_S8_GTPU_TEID %GTPV2_S2C_S5_S8_GTPU_TEID > %GTPV2_C2S_S5_S8_GTPU_IP %GTPV2_S2C_S5_S8_GTPU_IP %GTPV2_END_USER_IMSI > %GTPV2_END_USER_MSISDN %GTPV2_APN_NAME %GTPV2_ULI_MCC %GTPV2_ULI_MNC > %GTPV2_ULI_CELL_TAC %GTPV2_ULI_CELL_ID %GTPV2_RESPONSE_CAUSE %GTPV2_RAT_TYPE > %GTPV2_PDN_IP %GTPV2_END_USER_IMEI %GTPV2_C2S_S5_S8_GTPC_IP > %GTPV2_S2C_S5_S8_GTPC_IP %GTPV2_C2S_S5_S8_SGW_GTPU_TEID > %GTPV2_S2C_S5_S8_SGW_GTPU_TEID %GTPV2_C2S_S5_S8_SGW_GTPU_IP > %GTPV2_S2C_S5_S8_SGW_GTPU_IP %GTPV1_REQ_MSG_TYPE %GTPV1_RSP_MSG_TYPE > %GTPV1_C2S_TEID_DATA %GTPV1_C2S_TEID_CTRL %GTPV1_S2C_TEID_DATA > %GTPV1_S2C_TEID_CTRL %GTPV1_END_USER_IP %GTPV1_END_USER_IMSI > %GTPV1_END_USER_MSISDN %GTPV1_END_USER_IMEI %GTPV1_APN_NAME %GTPV1_RAT_TYPE > %GTPV1_RAI_MCC %GTPV1_RAI_MNC %GTPV1_RAI_LAC %GTPV1_RAI_RAC %GTPV1_ULI_MCC > %GTPV1_ULI_MNC %GTPV1_ULI_CELL_LAC %GTPV1_ULI_CELL_CI %GTPV1_ULI_SAC > %GTPV1_RESPONSE_CAUSE %SRC_FRAGMENTS %DST_FRAGMENTS %CLIENT_NW_LATENCY_MS > %SERVER_NW_LATENCY_MS %APPL_LATENCY_MS %RETRANSMITTED_IN_BYTES > %RETRANSMITTED_IN_PKTS %RETRANSMITTED_OUT_BYTES %RETRANSMITTED_OUT_PKTS > %OOORDER_IN_PKTS %OOORDER_OUT_PKTS %FLOW_ACTIVE_TIMEOUT > %FLOW_INACTIVE_TIMEOUT %MIN_TTL %MAX_TTL %IN_SRC_MAC %OUT_DST_MAC > %PACKET_SECTION_OFFSET %FRAME_LENGTH %SRC_TO_DST_MAX_THROUGHPUT > %SRC_TO_DST_MIN_THROUGHPUT %SRC_TO_DST_AVG_THROUGHPUT > %DST_TO_SRC_MAX_THROUGHPUT %DST_TO_SRC_MIN_THROUGHPUT > %DST_TO_SRC_AVG_THROUGHPUT %NUM_PKTS_UP_TO_128_BYTES > %NUM_PKTS_128_TO_256_BYTES %NUM_PKTS_256_TO_512_BYTES > %NUM_PKTS_512_TO_1024_BYTES %NUM_PKTS_1024_TO_1514_BYTES > %NUM_PKTS_OVER_1514_BYTES %LONGEST_FLOW_PKT %SHORTEST_FLOW_PKT > %NUM_PKTS_TTL_EQ_1 %NUM_PKTS_TTL_2_5 %NUM_PKTS_TTL_5_32 %NUM_PKTS_TTL_32_64 > %NUM_PKTS_TTL_64_96 %NUM_PKTS_TTL_96_128 %NUM_PKTS_TTL_128_160 > %NUM_PKTS_TTL_160_192 %NUM_PKTS_TTL_192_224 %NUM_PKTS_TTL_224_255 > %DURATION_IN %DURATION_OUT %TCP_WIN_MIN_IN %TCP_WIN_MAX_IN %TCP_WIN_MSS_IN > %TCP_WIN_SCALE_IN %TCP_WIN_MIN_OUT %TCP_WIN_MAX_OUT %TCP_WIN_MSS_OUT > %TCP_WIN_SCALE_OUT" > -A=/usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat > --city-list=/usr/share/ntopng/httpdocs/geoip/GeoLiteCity.dat > --kafka "127.0.0.1:9092;TEST" > -D=t > --tunnel > -b=1 > --smart-udp-frags > -f="udp" > > > -- nProbe traffic statistics: > > 19/Jan/2018 18:49:15 [nprobe.c:3106] Average traffic: [227.07 K pps][All > Traffic 1.18 Gb/sec][IP Traffic 1.05 Gb/sec][ratio 0.89] > 19/Jan/2018 18:49:15 [nprobe.c:3114] Current traffic: [222.87 K pps][1.14 > Gb/sec] > 19/Jan/2018 18:49:15 [nprobe.c:3120] Current flow export rate: [5350.3 > flows/sec] > 19/Jan/2018 18:49:15 [nprobe.c:3123] Flow drops: [export queue too > long=0][too many flows=0][ELK queue flow drops=0] > 19/Jan/2018 18:49:15 [nprobe.c:3128] Export Queue: 510035/4048000 [12.6 %] > 19/Jan/2018 18:49:15 [nprobe.c:3133] Flow Buckets: > [active=541656][allocated=1051691][toBeExported=510035] > 19/Jan/2018 18:49:15 [nprobe.c:3139] Kafka [flows exported=9071081/5350.3 > flows/sec][msgs sent=9071081/1.0 flows/msg][send errors=0] > 19/Jan/2018 18:49:15 [nprobe.c:2956] Processed packets: 388520559 (max bucket > search: 19) > 19/Jan/2018 18:49:15 [nprobe.c:2939] Fragment queue length: 0 > 19/Jan/2018 18:49:15 [nprobe.c:2962] WARNING: Your bucket search is too slow > (19): expect drops > 19/Jan/2018 18:49:15 [nprobe.c:2965] Flow export stats: [0 bytes/0 pkts][0 > flows/0 pkts sent] > 19/Jan/2018 18:49:15 [nprobe.c:2975] Flow drop stats: [1241630041 > bytes/273259031 pkts][0 flows] > 19/Jan/2018 18:49:15 [nprobe.c:2980] Total flow stats: [1241630041 > bytes/273259031 pkts][0 flows/0 pkts sent] > 19/Jan/2018 18:49:15 [nprobe.c:2991] Kafka [flows exported=9071083][msgs > sent=9071083/1.0 flows/msg][send errors=0] > > > > -- Our application statistics: > > 19/Jan/2018 18:49:15 [zcluster.c:173] Absolute Stats: Recv 5'604'403'857 pkts > (0 drops) - Forwarded 2'197'198'847 pkts (3'407'205'010 drops) > 19/Jan/2018 18:49:15 [zcluster.c:205] (In p2p1) RX 1176494979 pkts > Dropped 0 pkts (0.0 %) > 19/Jan/2018 18:49:15 [zcluster.c:205] (In p2p2) RX 4427908920 pkts > Dropped 0 pkts (0.0 %) > 19/Jan/2018 18:49:15 [zcluster.c:219] (Out Local) num:0 RX 2197166603 > pkts Dropped 3407205052 pkts (60.8 %) > 19/Jan/2018 18:49:15 [zcluster.c:239] Actual Stats: Recv 538'086.38 pps (0.00 > drops) - Forwarded 209'878.02 pps (328'208.35 drops) > > > > -- > Regards, > David Notivol > [email protected] <mailto:[email protected]> > > > > -- > Saludos, > David Notivol > [email protected] <mailto:[email protected]> > _______________________________________________ > Ntop-misc mailing list > [email protected] <mailto:[email protected]> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc > <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
_______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc
