Dear Mark, Also the layer-4 protocol is used when doing aggregations. As UDP is connectionless, there's no way for nProbe to determine the flow has terminated (there's no FIN packet) apart from using the idle timeout.
Simone > On 5 Jan 2018, at 14:54, Mark Petronic <[email protected]> wrote: > > Thank you, Simone, for the quick response. This very much clarifies it for > me. > > I have to say, as a new user of nprobe and noob to netflow, I am very > impressed with the timeliness and level of responses I get from this forum. > Very appreciated and helpful in me ramping up. > > As a follow up related to this answer, how does nprobe internally treat UDP > data in aggregations? For example, a UDP DNS request and corresponding > response. The 4-tuples would match up regarding src and dst ips and ports (at > least for the single request and response for one DNS request) but this is a > connectionless protocol. Does nprobe attempt to "finesse" these types of > paired packets into a common aggregation bin if the arrive within the window > of idle timeout or simply only treat TCP connections as ones that can be > aggregated? In my world, many of the tenant networks that we will be > monitoring have a lot of small TCP transactions and DNS UDP dominates much of > the flow volume - at least as a percentage of flow records (not necessarily > byte volume). > > Thanks... Mark > > On Thu, Jan 4, 2018 at 5:27 PM, Simone Mainardi <[email protected] > <mailto:[email protected]>> wrote: > Your NetFlow exporter outputs each bi-directional flow as two separate data > records -- one for the client-to-server direction and the other for the > server-to-client direction. Once the direction is given, one counter for the > bytes (i.e., IN_BYTES) is necessary. > > By default, nProbe automatically figures out and merges the two directions of > collected flows, turning the server-to-client IN_BYTES into the flow > OUT_BYTES. > > If you --disable-cache, you will prevent nProbe from doing any merge. In this > case, nProbe will act as a transparent proxy and will immediately output each > data record as-is, leaving the OUT_BYTES at zero. > Contrarily, if you don't --disable-cache, nProbe will automatically merge the > two data records into the same flow and then will only output one flow with > non-zero OUT_BYTES and IN_BYTES. > > > Simone > >> On 4 Jan 2018, at 20:33, Mark Petronic <[email protected] >> <mailto:[email protected]>> wrote: >> >> I am running nprobe as a collector, received flows from a few routers and >> forwarding to Kafka. I was doing some ingest performance testing and wanted >> to increase the rate of flow records/sec so I added --disable-cache to my >> configuration. This results in nearly 10X more records per second output >> from nprobe into Kafka which I wanted for testing. However, while testing >> throughput, I noticed that the OUT_BYTES field in the JSON records published >> to Kafka where always zero values now. If I remove --disable-cache, the >> OUT_BYTES values return to as expected (non-zero values)? Seems quite >> strange. Can anyone explain what is happening here? >> >> >> Welcome to nProbe v.8.2.171214 (r5982) for x86_64-unknown-linux-gnu >> with native PF_RING acceleration. >> Copyright 2002-17 ntop.org <http://ntop.org/> >> >> Build OS: CentOS Linux release 7.4.1708 (Core) >> SystemID: 68A2B43E76056A7E >> GIT rev: 8.2-stable:81448aa287d3968208054e69cdef2a10aa91758a:20171214 >> >> >> --disable-cache << Adding this config makes OUT_BYTES become zero for all >> flow records >> --interface=none >> --collector=none >> --collector-port=2055 >> --verbose=1 >> --lifetime-timeout=120 >> --idle-timeout=30 >> --queue-timeout=30 >> --flow-version=9 >> --hash-size=256000 >> --kafka-add-timestamp >> --kafka="kafka01:9092;netflow-raw;1" >> -T="%IPV4_SRC_ADDR %IPV4_DST_ADDR %L4_SRC_PORT %L4_DST_PORT %IPV4_SRC_MASK >> %IPV4_DST_MASK %IPV4_NEXT_HOP %IN_PKTS %IN_BYTES %OUT_PKTS %OUT_BYTES >> %FIRST_SWITCHED %LAST_SWITCHED %TCP_FLAGS %PROTOCOL %SRC_TOS %DIRECTION >> %EXPORTER_IPV4_ADDRESS %L4_SRC_PORT_MAP %L4_DST_PORT_MAP %INPUT_SNMP >> %OUTPUT_SNMP %FLOW_ID" >> >> _______________________________________________ >> Ntop-misc mailing list >> [email protected] <mailto:[email protected]> >> http://listgateway.unipi.it/mailman/listinfo/ntop-misc >> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> > > _______________________________________________ > Ntop-misc mailing list > [email protected] <mailto:[email protected]> > http://listgateway.unipi.it/mailman/listinfo/ntop-misc > <http://listgateway.unipi.it/mailman/listinfo/ntop-misc> > > _______________________________________________ > Ntop-misc mailing list > [email protected] > http://listgateway.unipi.it/mailman/listinfo/ntop-misc
_______________________________________________ Ntop-misc mailing list [email protected] http://listgateway.unipi.it/mailman/listinfo/ntop-misc
