Hi Mark, > On 9 Jan 2018, at 06:20, Mark Petronic <[email protected]> wrote: > > Thank you, Luca. In the help output, it indicates these aggregation fields: > > <VLAN Id>/<proto>/<IP>/<port>/<TOS>/<SCTP StreamId> > > We are not using VLANs in our network and we are not using SCTP. So, I assume > then that ONLY the following fields will be used for aggregations: > > <proto>/<IP>/<port>/<TOS> > > You said "Please pay attention to the nprobe startup log" but I do not see > anything here indicating anything about the aggregation bit mappings that are > in effect.
Better if you add -b 2 to see these messages. Before I further comment your email, let me clarify the meaning of aggregation in nProbe with an example. If you have asymmetric VLANs (i.e. A->B is sent over VLAN X, and B->A over VLAN Y) you need to put a 0 in the VLAN field (of -p) as otherwise A->B and B->A will be different flows and not the same bi-directional communication. > > I am concerned that nprobe, in our setup, is going to perform unintended > aggregates so I would have to design around that. Let me explain. We will be > acting as a network provider in a multi-tenant environment. Each tenant can > have overlapping private IP networks. I was thinking about sending flows from > many routers - across many tenants - to a single nprobe instance, as many as > that instance could handle, CPU-wise. I would stand up N instances then to > scale out to handle all tenant flows. Ok this means you are using nProbe as collector and not as probe > > I believe - as I understand the aggregation feature - that the following > could occur: > > Assume <proto>/<IP>/<port>/<TOS> is configured and TOS is constant and > protocol=TCP and, by coincidence, the connection 4-tuple just happens to > overlap between two tenant networks. > > Router A (running in tenant network X with EXPORTER_IPV4_ADDRESS 172.10.10.1) > sends a flow with tuples 10.2.3.4:5555 <http://10.2.3.4:5555/> -> > 10.6.7.8:443 <http://10.6.7.8:443/> > Router B (running in tenant network Y with EXPORTER_IPV4_ADDRESS 172.11.11.1) > sends a flow with tuples 10.2.3.4:5555 <http://10.2.3.4:5555/> -> > 10.6.7.8:443 <http://10.6.7.8:443/> > > Question: > > Even though these are coming from two different tenants (two different > routers), am I correct in concluding that these two flow records would be > aggregated in the same aggregation? I believe this is the case because the > EXPORTER_IPV4_ADDRESS is not part of the composite key used for aggregations. > I obviously don't want this to happen so I would have to design my collection > system to avoid this behavior if this is the case Your assumption is correct. To avoid that please add --disable-cache and it should work as expected Cheers Luca > > > > --interface=none > --collector=none > --collector-port=2055 > --verbose=1 > --lifetime-timeout=120 > --idle-timeout=30 > --queue-timeout=30 > --flow-version=9 > --hash-size=256000 > --kafka-add-timestamp > --kafka="kafka01:9092;netflow-raw;1" > -T="%IPV4_SRC_ADDR %IPV4_DST_ADDR %L4_SRC_PORT %L4_DST_PORT %IPV4_SRC_MASK > %IPV4_DST_MASK %IPV4_NEXT_HOP %IN_PKTS %IN_BYTES %OUT_PKTS %OUT_BYTES > %FIRST_SWITCHED %LAST_SWITCHED %TCP_FLAGS %PROTOCOL %SRC_TOS %DIRECTION > %EXPORTER_IPV4_ADDRESS" > > > 08/Jan/2018 18:08:42 [plugin.c:187] No plugins found in ./plugins > 08/Jan/2018 18:08:42 [plugin.c:195] Loading 23 plugins [.so] from > /usr/local/lib/nprobe/plugins > 08/Jan/2018 18:08:42 [nprobe.c:3784] ERROR: Invalid nProbe license > (/etc/nprobe.license) [Missing license file] > 08/Jan/2018 18:08:42 [nprobe.c:3791] ERROR: > ***************************************************** > 08/Jan/2018 18:08:42 [nprobe.c:3792] ERROR: ** > ** > 08/Jan/2018 18:08:42 [nprobe.c:3793] ERROR: ** Switching to DEMO MODE > (missing valid license) ** > 08/Jan/2018 18:08:42 [nprobe.c:3794] ERROR: ** > ** > 08/Jan/2018 18:08:42 [nprobe.c:3795] ERROR: ** Purchase your nProbe license > at ** > 08/Jan/2018 18:08:42 [nprobe.c:3796] ERROR: ** https://shop.ntop.org/ > <https://shop.ntop.org/> ** > 08/Jan/2018 18:08:42 [nprobe.c:3797] ERROR: ** > ** > 08/Jan/2018 18:08:42 [nprobe.c:3798] ERROR: > ***************************************************** > 08/Jan/2018 18:08:42 [nprobe.c:4809] WARNING: If you want to preserve the -M > value, please specify -w before -M > 08/Jan/2018 18:08:42 [nprobe.c:5755] WARNING: The output interfaceId is set > to 0: did you forget to use -Q perhaps ? > 08/Jan/2018 18:08:42 [nprobe.c:5758] WARNING: The input interfaceId is set to > 0: did you forget to use -u perhaps ? > 08/Jan/2018 18:08:42 [nprobe.c:5859] Welcome to nProbe v.8.2.171214 > ($Revision: 5982 $) for x86_64-unknown-linux-gnu with native PF_RING > acceleration > 08/Jan/2018 18:08:42 [nprobe.c:5869] Running on CentOS Linux release 7.4.1708 > (Core) > 08/Jan/2018 18:08:42 [nprobe.c:5880] [LICENSE] nProbe SystemId: > 68A2B43E76056A7E > 08/Jan/2018 18:08:42 [nprobe.c:5993] Sample rate [packet: 1][flow > collection/export: 1/1] > 08/Jan/2018 18:08:42 [nprobe.c:8432] ERROR: > *************************************************************** > 08/Jan/2018 18:08:42 [nprobe.c:8433] ERROR: * NOTE: This is a DEMO version > limited to 25000 flows export. * > 08/Jan/2018 18:08:42 [nprobe.c:8434] ERROR: > *************************************************************** > 08/Jan/2018 18:08:42 [exportPlugin.c:397] Trying to acquire metadata > information from kafka brokers. This could take several seconds. > 08/Jan/2018 18:08:42 [exportPlugin.c:413] Succesfully acquired metadata > information from broker(s) > 08/Jan/2018 18:08:42 [exportPlugin.c:425] 1 partions found > 08/Jan/2018 18:08:42 [nprobe.c:8440] Welcome to nProbe v.8.2.171214 for > x86_64-unknown-linux-gnu > 08/Jan/2018 18:08:42 [nprobe.c:7468] Using NetFlow Packet Payload Len: 1472 > 08/Jan/2018 18:08:42 [plugin.c:1155] 1 plugin(s) enabled > 08/Jan/2018 18:08:42 [nprobe.c:7907] Each flow is 98 bytes long > 08/Jan/2018 18:08:42 [nprobe.c:7908] The # flows per packet has been set to 14 > 08/Jan/2018 18:08:42 [nprobe.c:7911] IP TOS is accounted > 08/Jan/2018 18:08:42 [nprobe.c:7937] Non IPv4/v6 traffic is discarded > according to the template > 08/Jan/2018 18:08:42 [util.c:440] GeoIP: loaded AS config file > /usr/share/ntopng/httpdocs/geoip/GeoIPASNum.dat > 08/Jan/2018 18:08:42 [util.c:451] GeoIP: loaded AS IPv6 config file > /usr/share/ntopng/httpdocs/geoip/GeoIPASNumv6.dat > 08/Jan/2018 18:08:42 [nprobe.c:8772] Not capturing packet from interface > (collector mode) > 08/Jan/2018 18:08:42 [util.c:3591] nProbe changed user to 'nobody' > 08/Jan/2018 18:08:42 [plugin.c:900] Enabling plugin Export Plugin > 08/Jan/2018 18:08:42 [collect.c:144] Flow collector listening on port 2055 > (IPv4/v6) > 08/Jan/2018 18:08:42 [nprobe.c:8989] nProbe started successfully > 08/Jan/2018 18:08:43 [nprobe.c:3201] --------------------------------- > 08/Jan/2018 18:08:43 [nprobe.c:3202] Average traffic: [0.00 pps][All Traffic > 0 b/sec][IP Traffic 0 b/sec][ratio -nan] > 08/Jan/2018 18:08:43 [nprobe.c:3210] Current traffic: [0.00 pps][0 b/sec] > 08/Jan/2018 18:08:43 [nprobe.c:3216] Current flow export rate: [0.0 flows/sec] > 08/Jan/2018 18:08:43 [nprobe.c:3219] Flow drops: [export queue too > long=0][too many flows=0][ELK queue flow drops=0] > 08/Jan/2018 18:08:43 [nprobe.c:3224] Export Queue: 0/512000 [0.0 %] > 08/Jan/2018 18:08:43 [nprobe.c:3229] Flow Buckets: > [active=13395][allocated=13395][toBeExported=0] > 08/Jan/2018 18:08:43 [nprobe.c:3235] Kafka [flows exported=0/0.0 > flows/sec][msgs sent=0/0.0 flows/msg][send errors=0] > 08/Jan/2018 18:08:43 [nprobe.c:3260] Collector Threads: [757 pkts@0] > 08/Jan/2018 18:08:43 [nprobe.c:3052] Processed packets: 0 (max bucket search: > 7) > 08/Jan/2018 18:08:43 [nprobe.c:3035] Fragment queue length: 0 > 08/Jan/2018 18:08:43 [nprobe.c:3061] Flow export stats: [0 bytes/0 pkts][0 > flows/0 pkts sent] > 08/Jan/2018 18:08:43 [nprobe.c:3068] Flow collection: [collected pkts: > 757][processed flows: 20160] > 08/Jan/2018 18:08:43 [nprobe.c:3071] Flow drop stats: [0 bytes/0 pkts][0 > flows] > 08/Jan/2018 18:08:43 [nprobe.c:3076] Total flow stats: [0 bytes/0 pkts][0 > flows/0 pkts sent] > 08/Jan/2018 18:08:43 [nprobe.c:3087] Kafka [flows exported=0][msgs sent=0/0.0 > flows/msg][send errors=0] > 08/Jan/2018 18:09:13 [nprobe.c:3201] --------------------------------- > 08/Jan/2018 18:09:13 [nprobe.c:3202] Average traffic: [0.00 pps][All Traffic > 0 b/sec][IP Traffic 0 b/sec][ratio -nan] > 08/Jan/2018 18:09:13 [nprobe.c:3210] Current traffic: [0.00 pps][0 b/sec] > 08/Jan/2018 18:09:13 [nprobe.c:3216] Current flow export rate: [27.4 > flows/sec] > 08/Jan/2018 18:09:13 [nprobe.c:3219] Flow drops: [export queue too > long=0][too many flows=0][ELK queue flow drops=0] > 08/Jan/2018 18:09:13 [nprobe.c:3224] Export Queue: 0/512000 [0.0 %] > 08/Jan/2018 18:09:13 [nprobe.c:3229] Flow Buckets: > [active=167763][allocated=167763][toBeExported=0] > 08/Jan/2018 18:09:13 [nprobe.c:3235] Kafka [flows exported=822/27.4 > flows/sec][msgs sent=822/1.0 flows/msg][send errors=0] > 08/Jan/2018 18:09:13 [nprobe.c:3260] Collector Threads: [28566 pkts@0] > 08/Jan/2018 18:09:13 [nprobe.c:3052] Processed packets: 0 (max bucket search: > 8) > 08/Jan/2018 18:09:13 [nprobe.c:3035] Fragment queue length: 0 > 08/Jan/2018 18:09:13 [nprobe.c:3061] Flow export stats: [0 bytes/0 pkts][0 > flows/0 pkts sent] > 08/Jan/2018 18:09:13 [nprobe.c:3068] Flow collection: [collected pkts: > 28566][processed flows: 765143] > 08/Jan/2018 18:09:13 [nprobe.c:3071] Flow drop stats: [0 bytes/0 pkts][0 > flows] > 08/Jan/2018 18:09:13 [nprobe.c:3076] Total flow stats: [0 bytes/0 pkts][0 > flows/0 pkts sent] > 08/Jan/2018 18:09:13 [nprobe.c:3087] Kafka [flows exported=822][msgs > sent=822/1.0 flows/msg][send errors=0] > 08/Jan/2018 18:09:43 [nprobe.c:3201] --------------------------------- > 08/Jan/2018 18:09:43 [nprobe.c:3202] Average traffic: [0.00 pps][All Traffic > 0 b/sec][IP Traffic 0 b/sec][ratio -nan] > 08/Jan/2018 18:09:43 [nprobe.c:3210] Current traffic: [0.00 pps][0 b/sec] > 08/Jan/2018 18:09:43 [nprobe.c:3216] Current flow export rate: [4333.8 > flows/sec] > 08/Jan/2018 18:09:43 [nprobe.c:3219] Flow drops: [export queue too > long=0][too many flows=0][ELK queue flow drops=0] > 08/Jan/2018 18:09:43 [nprobe.c:3224] Export Queue: 0/512000 [0.0 %] > 08/Jan/2018 18:09:43 [nprobe.c:3229] Flow Buckets: > [active=96146][allocated=96146][toBeExported=0] > 08/Jan/2018 18:09:43 [nprobe.c:3235] Kafka [flows exported=130835/4333.8 > flows/sec][msgs sent=130835/1.0 flows/msg][send errors=0] > 08/Jan/2018 18:09:43 [nprobe.c:3260] Collector Threads: [50988 pkts@0] > 08/Jan/2018 18:09:43 [nprobe.c:3052] Processed packets: 0 (max bucket search: > 8) > 08/Jan/2018 18:09:43 [nprobe.c:3035] Fragment queue length: 0 > 08/Jan/2018 18:09:43 [nprobe.c:3061] Flow export stats: [0 bytes/0 pkts][0 > flows/0 pkts sent] > 08/Jan/2018 18:09:43 [nprobe.c:3068] Flow collection: [collected pkts: > 50988][processed flows: 1376945] > 08/Jan/2018 18:09:43 [nprobe.c:3071] Flow drop stats: [0 bytes/0 pkts][0 > flows] > 08/Jan/2018 18:09:43 [nprobe.c:3076] Total flow stats: [0 bytes/0 pkts][0 > flows/0 pkts sent] > 08/Jan/2018 18:09:43 [nprobe.c:3087] Kafka [flows exported=130835][msgs > sent=130835/1.0 flows/msg][send errors=0] > > > > On Mon, Jan 8, 2018 at 1:10 PM, Luca Deri <[email protected] > <mailto:[email protected]>> wrote: > Mark > the default is 1/1/1/1/1/1 but please note that depending on the template > some fields will be set to 0. Please pay attention to the nprobe startup log > > Thanks Luca > >> On 8 Jan 2018, at 19:01, Mark Petronic <[email protected] >> <mailto:[email protected]>> wrote: >> >> Some indicate the default in the -h output and some do not. Can some please >> tell me the default value for --aggregation in v8.2? Thank you! >> _______________________________________________ >> Ntop-misc mailing list >> [email protected] <mailto:[email protected]> >> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> > > _______________________________________________ > Ntop-misc mailing list > [email protected] <mailto:[email protected]> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc > <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> > > _______________________________________________ > Ntop-misc mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop-misc
_______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc
