Re: [Ntop-misc] nProbe big log file with elastic search

[Luca Deri]

Ohad
I am unable to see such file on my setup. Can you please send me a
portion of this log? Are you sure nprobe is creating it?

Regards Luca

On 02/12/2015 08:47, Ohad Kleinman wrote:
>
> Hi Luca,
>
> The log file that I am referring to is nprobe-e...@0.log
> <mailto:nprobe-e...@0.log> located in /var/log/nprobe directory.
>
>  
>
> Yes currently we are using both dumping files in text format and also
> to the elastic search.
>
>  
>
> I hope this helps.
>
>  
>
> Ohad
>
>  
>
> *From:*ntop-misc-boun...@listgateway.unipi.it
> <mailto:ntop-misc-boun...@listgateway.unipi.it>
> [mailto:ntop-misc-boun...@listgateway.unipi.it
> <mailto:ntop-misc-boun...@listgateway.unipi.it>] *On Behalf Of *Luca Deri
> *Sent:* Wednesday, December 02, 2015 9:37 AM
> *To:* ntop-misc@listgateway.unipi.it
> <mailto:ntop-misc@listgateway.unipi.it>
> *Subject:* Re: [Ntop-misc] nProbe big log file with elastic search
>
>  
>
> Hi Ohad,
>
> using the latest nProbe I have been unable to reproduce the issue you
> reported. I have even started nprobe with strace just to make sure I
> didn’t miss anything but the .log file you mention is not created.
>
>  
>
> Instead using -P you are telling nprobe to dump flows in text format
> (in addition to pushing them to ES): is this what you want? 
>
>  
>
> Where is this .log file created? (path I mean)
>
>  
>
> Regards Luca
>
>  
>
> On 29 Nov 2015, at 13:04, Ohad Kleinman <oh...@vglnt.com
> <mailto:oh...@vglnt.com>> wrote:
>
>  
>
> Luca,
>
> Please see attached the configuration file that we are using.
>
>  
>
> Ohad
>
>  
>
> *From:* ntop-misc-boun...@listgateway.unipi.it
> <mailto:ntop-misc-boun...@listgateway.unipi.it> 
> [mailto:ntop-misc-boun...@listgateway.unipi.it
> <mailto:ntop-misc-boun...@listgateway.unipi.it>] *On Behalf
> Of *Luca Deri
> *Sent:* Sunday, November 29, 2015 12:22 PM
> *To:* ntop-misc@listgateway.unipi.it
> <mailto:ntop-misc@listgateway.unipi.it>
> *Subject:* Re: [Ntop-misc] nProbe big log file with elastic search
>
>  
>
> Ohad,
>
> nProbe should not write to this log. I think it is a combination
> of options we do not handle properly. Can you please let send me
> the complete command line you are using to start nProbe so I can
> analyse it?
>
>  
>
> Thanks Luca
>
>  
>
> On 29 Nov 2015, at 08:14, Ohad Kleinman <oh...@vglnt.com
> <mailto:oh...@vglnt.com>> wrote:
>
>  
>
> Hi Luca,
>
> Can you confirm if there is a way to make the nprobe
> to *not* write to the log file each flow that is being
> exported to elastic?
>
>  
>
> Thanks
>
>  
>
> Ohad
>
>  
>
>  
>
> *From:* ntop-misc-boun...@listgateway.unipi.it
>     <mailto:ntop-misc-boun...@listgateway.unipi.it> 
> [mailto:ntop-misc-boun...@listgateway.unipi.it
> <mailto:ntop-misc-boun...@listgateway.unipi.it>] *On Behalf
> Of *Luca Deri
> *Sent:* Monday, November 23, 2015 2:59 PM
> *To:* ntop-misc@listgateway.unipi.it
> <mailto:ntop-misc@listgateway.unipi.it>
> *Subject:* Re: [Ntop-misc] nProbe big log file with elastic search
>
>  
>
> Hi Ohad,
>
> is this file on the ElasticSearch side right?
>
>  
>
> Regards Luca
>
>  
>
> On 18 Nov 2015, at 15:34, Ohad Kleinman <oh...@vglnt.com
> <mailto:oh...@vglnt.com>> wrote:
>
>  
>
> Hi,
>
> We are using nProbe with the option of writing all flows
> into elastic search, the nprobe-e...@0.log
> <mailto:nprobe-e...@0.log> file is becoming large as each
> flow that is written into the elastic search is also being
> written into the log file.
>
>  
>
> Is there any parameter that can disable this?
>
>  
>
> Thanks.
>
>  
>
> Ohad
>
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it
> <mailto:Ntop-misc@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc
>
>  
>
> __

Re: [Ntop-misc] nProbe big log file with elastic search

[Ohad Kleinman]

Luca,

Please see a link to the attached the log file, I hope this helps.



https://dl.dropboxusercontent.com/u/27973370/nprobe-eth0%400.log





Ohad







*From:* ntop-misc-boun...@listgateway.unipi.it [mailto:
ntop-misc-boun...@listgateway.unipi.it] *On Behalf Of *Luca Deri
*Sent:* Wednesday, December 02, 2015 2:42 PM
*To:* ntop-misc@listgateway.unipi.it
*Subject:* Re: [Ntop-misc] nProbe big log file with elastic search



Ohad
I am unable to see such file on my setup. Can you please send me a portion
of this log? Are you sure nprobe is creating it?

Regards Luca

On 02/12/2015 08:47, Ohad Kleinman wrote:

Hi Luca,

The log file that I am referring to is nprobe-e...@0.log located in
/var/log/nprobe directory.



Yes currently we are using both dumping files in text format and also to
the elastic search.



I hope this helps.



Ohad



*From:* ntop-misc-boun...@listgateway.unipi.it [mailto:
ntop-misc-boun...@listgateway.unipi.it] *On Behalf Of *Luca Deri
*Sent:* Wednesday, December 02, 2015 9:37 AM
*To:* ntop-misc@listgateway.unipi.it
*Subject:* Re: [Ntop-misc] nProbe big log file with elastic search



Hi Ohad,

using the latest nProbe I have been unable to reproduce the issue you
reported. I have even started nprobe with strace just to make sure I didn’t
miss anything but the .log file you mention is not created.



Instead using -P you are telling nprobe to dump flows in text format (in
addition to pushing them to ES): is this what you want?



Where is this .log file created? (path I mean)



Regards Luca



On 29 Nov 2015, at 13:04, Ohad Kleinman <oh...@vglnt.com> wrote:



Luca,

Please see attached the configuration file that we are using.



Ohad



*From:* ntop-misc-boun...@listgateway.unipi.it [mailto:
ntop-misc-boun...@listgateway.unipi.it] *On Behalf Of *Luca Deri
*Sent:* Sunday, November 29, 2015 12:22 PM
*To:* ntop-misc@listgateway.unipi.it
*Subject:* Re: [Ntop-misc] nProbe big log file with elastic search



Ohad,

nProbe should not write to this log. I think it is a combination of options
we do not handle properly. Can you please let send me the complete command
line you are using to start nProbe so I can analyse it?



Thanks Luca



On 29 Nov 2015, at 08:14, Ohad Kleinman <oh...@vglnt.com> wrote:



Hi Luca,

Can you confirm if there is a way to make the nprobe to *not* write to the
log file each flow that is being exported to elastic?



Thanks



Ohad





*From:* ntop-misc-boun...@listgateway.unipi.it [mailto:
ntop-misc-boun...@listgateway.unipi.it] *On Behalf Of *Luca Deri
*Sent:* Monday, November 23, 2015 2:59 PM
*To:* ntop-misc@listgateway.unipi.it
*Subject:* Re: [Ntop-misc] nProbe big log file with elastic search



Hi Ohad,

is this file on the ElasticSearch side right?



Regards Luca



On 18 Nov 2015, at 15:34, Ohad Kleinman <oh...@vglnt.com> wrote:



Hi,

We are using nProbe with the option of writing all flows into elastic
search, the nprobe-e...@0.log file is becoming large as each flow that is
written into the elastic search is also being written into the log file.



Is there any parameter that can disable this?



Thanks.



Ohad

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc



___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc



___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc






___

Ntop-misc mailing list

Ntop-misc@listgateway.unipi.it

http://listgateway.unipi.it/mailman/listinfo/ntop-misc
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nProbe big log file with elastic search

[Luca Deri]

Hi Ohad,
using the latest nProbe I have been unable to reproduce the issue you reported. 
I have even started nprobe with strace just to make sure I didn’t miss anything 
but the .log file you mention is not created.

Instead using -P you are telling nprobe to dump flows in text format (in 
addition to pushing them to ES): is this what you want? 

Where is this .log file created? (path I mean)

Regards Luca

> On 29 Nov 2015, at 13:04, Ohad Kleinman <oh...@vglnt.com> wrote:
> 
> Luca,
> Please see attached the configuration file that we are using.
>  
> Ohad
>  
> From: ntop-misc-boun...@listgateway.unipi.it 
> <mailto:ntop-misc-boun...@listgateway.unipi.it> 
> [mailto:ntop-misc-boun...@listgateway.unipi.it 
> <mailto:ntop-misc-boun...@listgateway.unipi.it>] On Behalf Of Luca Deri
> Sent: Sunday, November 29, 2015 12:22 PM
> To: ntop-misc@listgateway.unipi.it <mailto:ntop-misc@listgateway.unipi.it>
> Subject: Re: [Ntop-misc] nProbe big log file with elastic search
>  
> Ohad,
> nProbe should not write to this log. I think it is a combination of options 
> we do not handle properly. Can you please let send me the complete command 
> line you are using to start nProbe so I can analyse it?
>  
> Thanks Luca
>  
>> On 29 Nov 2015, at 08:14, Ohad Kleinman <oh...@vglnt.com 
>> <mailto:oh...@vglnt.com>> wrote:
>>  
>> Hi Luca,
>> Can you confirm if there is a way to make the nprobe to not write to the log 
>> file each flow that is being exported to elastic?
>>  
>> Thanks
>>  
>> Ohad
>>  
>>  
>> From: ntop-misc-boun...@listgateway.unipi.it 
>> <mailto:ntop-misc-boun...@listgateway.unipi.it> 
>> [mailto:ntop-misc-boun...@listgateway.unipi.it 
>> <mailto:ntop-misc-boun...@listgateway.unipi.it>] On Behalf Of Luca Deri
>> Sent: Monday, November 23, 2015 2:59 PM
>> To: ntop-misc@listgateway.unipi.it <mailto:ntop-misc@listgateway.unipi.it>
>> Subject: Re: [Ntop-misc] nProbe big log file with elastic search
>>  
>> Hi Ohad,
>> is this file on the ElasticSearch side right?
>>  
>> Regards Luca
>>  
>>> On 18 Nov 2015, at 15:34, Ohad Kleinman <oh...@vglnt.com 
>>> <mailto:oh...@vglnt.com>> wrote:
>>>  
>>> Hi,
>>> We are using nProbe with the option of writing all flows into elastic 
>>> search, the nprobe-e...@0.log <mailto:nprobe-e...@0.log> file is becoming 
>>> large as each flow that is written into the elastic search is also being 
>>> written into the log file.
>>>  
>>> Is there any parameter that can disable this?
>>>  
>>> Thanks.
>>>  
>>> Ohad
>>> ___
>>> Ntop-misc mailing list
>>> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
>>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
>>> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
>>  
>> ___
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
>> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
>  
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nProbe big log file with elastic search

[Ohad Kleinman]

Hi Luca,

The log file that I am referring to is nprobe-e...@0.log located in
/var/log/nprobe directory.



Yes currently we are using both dumping files in text format and also to
the elastic search.



I hope this helps.



Ohad



*From:* ntop-misc-boun...@listgateway.unipi.it [mailto:
ntop-misc-boun...@listgateway.unipi.it] *On Behalf Of *Luca Deri
*Sent:* Wednesday, December 02, 2015 9:37 AM
*To:* ntop-misc@listgateway.unipi.it
*Subject:* Re: [Ntop-misc] nProbe big log file with elastic search



Hi Ohad,

using the latest nProbe I have been unable to reproduce the issue you
reported. I have even started nprobe with strace just to make sure I didn’t
miss anything but the .log file you mention is not created.



Instead using -P you are telling nprobe to dump flows in text format (in
addition to pushing them to ES): is this what you want?



Where is this .log file created? (path I mean)



Regards Luca



On 29 Nov 2015, at 13:04, Ohad Kleinman <oh...@vglnt.com> wrote:



Luca,

Please see attached the configuration file that we are using.



Ohad



*From:* ntop-misc-boun...@listgateway.unipi.it [mailto:
ntop-misc-boun...@listgateway.unipi.it] *On Behalf Of *Luca Deri
*Sent:* Sunday, November 29, 2015 12:22 PM
*To:* ntop-misc@listgateway.unipi.it
*Subject:* Re: [Ntop-misc] nProbe big log file with elastic search



Ohad,

nProbe should not write to this log. I think it is a combination of options
we do not handle properly. Can you please let send me the complete command
line you are using to start nProbe so I can analyse it?



Thanks Luca



On 29 Nov 2015, at 08:14, Ohad Kleinman <oh...@vglnt.com> wrote:



Hi Luca,

Can you confirm if there is a way to make the nprobe to *not* write to the
log file each flow that is being exported to elastic?



Thanks



Ohad





*From:* ntop-misc-boun...@listgateway.unipi.it [mailto:
ntop-misc-boun...@listgateway.unipi.it] *On Behalf Of *Luca Deri
*Sent:* Monday, November 23, 2015 2:59 PM
*To:* ntop-misc@listgateway.unipi.it
*Subject:* Re: [Ntop-misc] nProbe big log file with elastic search



Hi Ohad,

is this file on the ElasticSearch side right?



Regards Luca



On 18 Nov 2015, at 15:34, Ohad Kleinman <oh...@vglnt.com> wrote:



Hi,

We are using nProbe with the option of writing all flows into elastic
search, the nprobe-e...@0.log file is becoming large as each flow that is
written into the elastic search is also being written into the log file.



Is there any parameter that can disable this?



Thanks.



Ohad

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc



___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc



___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nProbe big log file with elastic search

[Luca Deri]

Ohad,
nProbe should not write to this log. I think it is a combination of options we 
do not handle properly. Can you please let send me the complete command line 
you are using to start nProbe so I can analyse it?

Thanks Luca

> On 29 Nov 2015, at 08:14, Ohad Kleinman <oh...@vglnt.com> wrote:
> 
> Hi Luca,
> Can you confirm if there is a way to make the nprobe to not write to the log 
> file each flow that is being exported to elastic?
>  
> Thanks
>  
> Ohad
>  
>  
> From: ntop-misc-boun...@listgateway.unipi.it 
> <mailto:ntop-misc-boun...@listgateway.unipi.it> 
> [mailto:ntop-misc-boun...@listgateway.unipi.it 
> <mailto:ntop-misc-boun...@listgateway.unipi.it>] On Behalf Of Luca Deri
> Sent: Monday, November 23, 2015 2:59 PM
> To: ntop-misc@listgateway.unipi.it <mailto:ntop-misc@listgateway.unipi.it>
> Subject: Re: [Ntop-misc] nProbe big log file with elastic search
>  
> Hi Ohad,
> is this file on the ElasticSearch side right?
>  
> Regards Luca
>  
>> On 18 Nov 2015, at 15:34, Ohad Kleinman <oh...@vglnt.com 
>> <mailto:oh...@vglnt.com>> wrote:
>>  
>> Hi,
>> We are using nProbe with the option of writing all flows into elastic 
>> search, the nprobe-e...@0.log <mailto:nprobe-e...@0.log> file is becoming 
>> large as each flow that is written into the elastic search is also being 
>> written into the log file.
>>  
>> Is there any parameter that can disable this?
>>  
>> Thanks.
>>  
>> Ohad
>> ___
>> Ntop-misc mailing list
>> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
>> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
>> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
>  
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it <mailto:Ntop-misc@listgateway.unipi.it>
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> <http://listgateway.unipi.it/mailman/listinfo/ntop-misc>
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nProbe big log file with elastic search

[Ohad Kleinman]

Hi Luca,

Can you confirm if there is a way to make the nprobe to *not* write to the
log file each flow that is being exported to elastic?



Thanks



Ohad





*From:* ntop-misc-boun...@listgateway.unipi.it [mailto:
ntop-misc-boun...@listgateway.unipi.it] *On Behalf Of *Luca Deri
*Sent:* Monday, November 23, 2015 2:59 PM
*To:* ntop-misc@listgateway.unipi.it
*Subject:* Re: [Ntop-misc] nProbe big log file with elastic search



Hi Ohad,

is this file on the ElasticSearch side right?



Regards Luca



On 18 Nov 2015, at 15:34, Ohad Kleinman <oh...@vglnt.com> wrote:



Hi,

We are using nProbe with the option of writing all flows into elastic
search, the nprobe-e...@0.log file is becoming large as each flow that is
written into the elastic search is also being written into the log file.



Is there any parameter that can disable this?



Thanks.



Ohad

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nProbe big log file with elastic search

[Ohad Kleinman]

Hi Luca,

This file is in the var/log/nprobe folder, it contains many lines like this:



{"index": {"_type": "nProbe", "_index": "nprobe"}}

{"IPV4_SRC_ADDR":"10.0.5.125","IPV4_DST_ADDR":"10.0.4.24","IPV4_NEXT_HOP":"0.0.0.0","INPUT_SNMP":65535,"OUTPUT_SNMP":65535,"IN_PKTS":66,"IN_BYTES":18430,"FIRST_SWITCHED":1448283463,"LAST_SWITCHED":1448283522,"L4_SRC_PORT":2555,"L4_DST_PORT":60299,"TCP_FLAGS":24,"PROTOCOL":6,"SRC_TOS":0,"SRC_AS":0,"DST_AS":0,"IPV4_SRC_MASK":0,"IPV4_DST_MASK":0,"@version":"1","@timestamp":"2015-11-23T12:58:44Z",
"EXPORTER_IPV4_ADDRESS":"10.0.1.64"}





Regards,



Ohad



*From:* ntop-misc-boun...@listgateway.unipi.it [mailto:
ntop-misc-boun...@listgateway.unipi.it] *On Behalf Of *Luca Deri
*Sent:* Monday, November 23, 2015 2:59 PM
*To:* ntop-misc@listgateway.unipi.it
*Subject:* Re: [Ntop-misc] nProbe big log file with elastic search



Hi Ohad,

is this file on the ElasticSearch side right?



Regards Luca



On 18 Nov 2015, at 15:34, Ohad Kleinman <oh...@vglnt.com> wrote:



Hi,

We are using nProbe with the option of writing all flows into elastic
search, the nprobe-e...@0.log file is becoming large as each flow that is
written into the elastic search is also being written into the log file.



Is there any parameter that can disable this?



Thanks.



Ohad

___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

Re: [Ntop-misc] nProbe big log file with elastic search

[Luca Deri]

Hi Ohad,
is this file on the ElasticSearch side right?

Regards Luca

> On 18 Nov 2015, at 15:34, Ohad Kleinman  wrote:
> 
> Hi,
> We are using nProbe with the option of writing all flows into elastic search, 
> the nprobe-e...@0.log  file is becoming large as 
> each flow that is written into the elastic search is also being written into 
> the log file.
>  
> Is there any parameter that can disable this?
>  
> Thanks.
>  
> Ohad
> ___
> Ntop-misc mailing list
> Ntop-misc@listgateway.unipi.it 
> http://listgateway.unipi.it/mailman/listinfo/ntop-misc 
> 
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc

[Ntop-misc] nProbe big log file with elastic search

[Ohad Kleinman]

Hi,

We are using nProbe with the option of writing all flows into elastic
search, the nprobe-e...@0.log file is becoming large as each flow that is
written into the elastic search is also being written into the log file.



Is there any parameter that can disable this?



Thanks.



Ohad
___
Ntop-misc mailing list
Ntop-misc@listgateway.unipi.it
http://listgateway.unipi.it/mailman/listinfo/ntop-misc