Hi Paolo, thanks for your answer. I managed to fix the issue with exactly your hints. I set sql_cache_entries to 64k and added a id field to my custom sql scheme.
For completion of data my aggregate line looks like: aggregate: vlan, src_host, dst_host, src_port, dst_port, proto In between I encounterred warnings like this: Nov 15 13:51:10 WARN ( default/memory ): Unable to allocate more memory pools, clear stats manually! But isn't this one an error? When pmacct is not able to allocate memory for more pools data gets lost, or not? I ask this question because I wrote a small grep check script to monitor ERROR lines in the log file. Then I saw these lines but next to them lines like these: Nov 15 14:02:06 WARN ( default/core ): Abnormal exit status detected for child PID 12614 Nov 15 14:02:07 WARN ( default/core ): eth0: no IPv4 address assigned So a WARN message for an abnormal exit is ok when pmacct was killed by my init script and a WARN for my eth0 interface has no ip address because it's bridged and the bridge has the ip is ok, too. Maybe it's possible to change the severity of the allocate memory message to ERROR in one of the next releases. Nevertheless, this needs to be said: Paolo, you did really great work. Really cool software and thank you for your great support over years. cheers Andi Am 16.11.2015 um 10:27 schrieb Paolo Lucente: > Hi Andreas, > > The issue should not be connected at all to plugin_pipe_size and > plugin_buffer_size sizes - did you find a link between the issue > and these config directives somewhere in the archives? > > This may be more connected to sql_cache_entries (although you seem > to have it configured already large enough). You did not show the > 'aggregate' line of your config - can you please post it? > > A way to troubleshoot (or tackle) the issue is to add an id field > to the SQL table structure like described here: > > http://wiki.pmacct.net/CustomizingTheSqlIndexes > > This will prevent the current error message and, summarizing data > without the id field should make evident what behaviour duplicate > entries are following, ie. whether a specific field is responsible > for this. > > Cheers, > Paolo > > On Sat, Nov 14, 2015 at 04:10:39PM +0100, Andreas Nitsche wrote: >> Hi, >> >> I've already found some discussions about the duplicate entry problem >> and tried to adjust plugin_pipe_size (5 GB) and plugin_buffer_size (50 MB). >> >> Nov 14 14:35:19 ERROR ( default/mysql ): Duplicate entry >> '0----0-0-0-212.224.83.112-212.224.83.74-54644-4949-tcp-0-2015-11' for >> key 'PRIMARY' >> >> This is the main part of my pmacctd.conf configuration: >> >> ... >> ! >> ! MEMORY CONFIG >> ! >> imt_path: /tmp/pmacct.pipe.eth0 >> ! echo 5*1024*1024*1024 | bc -l (5 GB) >> plugin_pipe_size: 5368709120 >> ! echo 50*1024*1024 | bc -l (50 MB) >> plugin_buffer_size: 52428800 >> >> !plugins: mysql >> ... >> sql_table_version: 7 >> ... >> sql_dont_try_update: true >> sql_optimize_clauses: true >> sql_refresh_time: 60 >> sql_history: 1m >> sql_history_roundoff: m >> sql_cache_entries: 10000000 >> ... >> >> I'm writing the data in monthly divided tables where a table is from 3.6 >> to 7.4 GiB large. >> >> Could you give me some hints to get rid of this error? What are the >> largest values I should try? Is there any possibility to check the size >> of the memory table while pmacctd is running? >> >> thanks in advance >> Andi >> >> _______________________________________________ >> pmacct-discussion mailing list >> http://www.pmacct.net/#mailinglists > > _______________________________________________ > pmacct-discussion mailing list > http://www.pmacct.net/#mailinglists > -- sixhop.net sixhop.net Andreas Nitsche Pucher Str. 21-23 82256 Fürstenfeldbruck Fon: 08141 31 597-0 Fax: 08141 31 597-1 E-Mail: [email protected] <mailto:[email protected]> Web: www.sixhop.net <http://www.sixhop.net/> _______________________________________________ pmacct-discussion mailing list http://www.pmacct.net/#mailinglists
