Hi Andreas, Thanks a lot for the kind words.
An error condition would in principle be one that prevents the software from running (and hence each error message is currently followed by an exit() call): no interface to bind to, no memory available, etc.. The condition you are running into is pmacct (by default) does limit the own memory allocation; you can configure it to grow indefinitely in memory with "imt_mem_pools_number: 0"; it looks correct to me this condition is not followed by an exit() call and hence is marked as a warning. Cheers, Paolo On Mon, Nov 16, 2015 at 10:36:58AM +0100, Andreas Nitsche wrote: > Hi Paolo, > > thanks for your answer. I managed to fix the issue with exactly your > hints. I set sql_cache_entries to 64k and added a id field to my custom > sql scheme. > > For completion of data my aggregate line looks like: > > aggregate: vlan, src_host, dst_host, src_port, dst_port, proto > > In between I encounterred warnings like this: > > Nov 15 13:51:10 WARN ( default/memory ): Unable to allocate more memory > pools, clear stats manually! > > But isn't this one an error? When pmacct is not able to allocate memory > for more pools data gets lost, or not? > > I ask this question because I wrote a small grep check script to monitor > ERROR lines in the log file. Then I saw these lines but next to them > lines like these: > > Nov 15 14:02:06 WARN ( default/core ): Abnormal exit status detected for > child PID 12614 > Nov 15 14:02:07 WARN ( default/core ): eth0: no IPv4 address assigned > > So a WARN message for an abnormal exit is ok when pmacct was killed by > my init script and a WARN for my eth0 interface has no ip address > because it's bridged and the bridge has the ip is ok, too. > > Maybe it's possible to change the severity of the allocate memory > message to ERROR in one of the next releases. > > Nevertheless, this needs to be said: Paolo, you did really great work. > Really cool software and thank you for your great support over years. > > cheers > Andi > > Am 16.11.2015 um 10:27 schrieb Paolo Lucente: > > Hi Andreas, > > > > The issue should not be connected at all to plugin_pipe_size and > > plugin_buffer_size sizes - did you find a link between the issue > > and these config directives somewhere in the archives? > > > > This may be more connected to sql_cache_entries (although you seem > > to have it configured already large enough). You did not show the > > 'aggregate' line of your config - can you please post it? > > > > A way to troubleshoot (or tackle) the issue is to add an id field > > to the SQL table structure like described here: > > > > http://wiki.pmacct.net/CustomizingTheSqlIndexes > > > > This will prevent the current error message and, summarizing data > > without the id field should make evident what behaviour duplicate > > entries are following, ie. whether a specific field is responsible > > for this. > > > > Cheers, > > Paolo > > > > On Sat, Nov 14, 2015 at 04:10:39PM +0100, Andreas Nitsche wrote: > >> Hi, > >> > >> I've already found some discussions about the duplicate entry problem > >> and tried to adjust plugin_pipe_size (5 GB) and plugin_buffer_size (50 MB). > >> > >> Nov 14 14:35:19 ERROR ( default/mysql ): Duplicate entry > >> '0----0-0-0-212.224.83.112-212.224.83.74-54644-4949-tcp-0-2015-11' for > >> key 'PRIMARY' > >> > >> This is the main part of my pmacctd.conf configuration: > >> > >> ... > >> ! > >> ! MEMORY CONFIG > >> ! > >> imt_path: /tmp/pmacct.pipe.eth0 > >> ! echo 5*1024*1024*1024 | bc -l (5 GB) > >> plugin_pipe_size: 5368709120 > >> ! echo 50*1024*1024 | bc -l (50 MB) > >> plugin_buffer_size: 52428800 > >> > >> !plugins: mysql > >> ... > >> sql_table_version: 7 > >> ... > >> sql_dont_try_update: true > >> sql_optimize_clauses: true > >> sql_refresh_time: 60 > >> sql_history: 1m > >> sql_history_roundoff: m > >> sql_cache_entries: 10000000 > >> ... > >> > >> I'm writing the data in monthly divided tables where a table is from 3.6 > >> to 7.4 GiB large. > >> > >> Could you give me some hints to get rid of this error? What are the > >> largest values I should try? Is there any possibility to check the size > >> of the memory table while pmacctd is running? > >> > >> thanks in advance > >> Andi > >> > >> _______________________________________________ > >> pmacct-discussion mailing list > >> http://www.pmacct.net/#mailinglists > > > > _______________________________________________ > > pmacct-discussion mailing list > > http://www.pmacct.net/#mailinglists > > > > -- > sixhop.net > > sixhop.net > Andreas Nitsche > Pucher Str. 21-23 > 82256 Fürstenfeldbruck > > Fon: 08141 31 597-0 > Fax: 08141 31 597-1 > E-Mail: [email protected] <mailto:[email protected]> > Web: www.sixhop.net <http://www.sixhop.net/> > > > _______________________________________________ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
