Hi Steffen, The latest SNMP4J-3.0.0-SNAPSHOT contains a getTable method in TableUtils with a new parameter SparseTableMode which can be used to control the behaviour for tables that do not contain Null (not-accessible) cells (= dense tables).
Even for the situation, that rows appear or disappear during the retrieval, a solution could be implemented. In this case, a GET will fetch the missing columns. If they cannot be fetched, the row is ignored (because it disappeared). Hope this helps? Best regards, Frank > On 12. Jul 2018, at 08:40, Frank Fock <f...@agentpp.com> wrote: > > Hi Steffen, > > If the agent sends a tooBig error on a GETBULK request, then this is an error > in the agent. See RFC3416 4.2.3: > > If the size of the message encapsulating the Response-PDU > containing the requested number of variable bindings would be > greater than either a local constraint or the maximum message > size of the originator, then the response is generated with a > lesser number of variable bindings. This lesser number is the > ordered set of variable bindings with some of the variable > bindings at the end of the set removed, such that the size of > the message encapsulating the Response-PDU is approximately > equal to but no greater than either a local constraint or the > maximum message size of the originator. Note that the number > of variable bindings removed has no relationship to the values > of N, M, or R. > > For the issue you reported, there is no general solution, because it > interferes with sparse tables. > A solution would either decrease the performance for sparse tables or will > filter out sparse rows. > The latter is not acceptable for intentionally sparse tables. > For dense tables, the filtering could be the best option. Although it would > hide new rows although the command generator already detected them. > > I am currently about to add an option for getDenseTable to activate a > filtering for new rows that appear during the table retrieval and are > therefore incompletely received. Would that help you? > > Best regards, > Frank > >> On 9. Jul 2018, at 19:45, Steffen Brüntjen <steffen.bruent...@macmon.eu> >> wrote: >> >> Hi Frank >> >> Thank you for having a look at it. I agree, the performance with many >> bindings is indeed *much* higher and yes, values should be retrieved >> row-by-row in order to avoid data inconsistencies. But there are also >> problems with many bindings: >> >> 1. Since the agent can not - in the contrast to max-repetition-count - >> decide how many values to send, the packet size might get too big if you >> have a table with many (big) columns. >> >> 2. There are agents that get into trouble when many columns are requested. >> This often results in timeouts (no tooBig error) and then there's no other >> option to requesting fewer bindings. >> >> Maybe the proposed change is the way to go, it's decent, but effective (I >> believe). >> >> Best regards >> Steffen >> >> >> -----Original Message----- >> From: Frank Fock [mailto:f...@agentpp.com] >> Sent: Freitag, 6. Juli 2018 18:55 >> To: Steffen Brüntjen <steffen.bruent...@macmon.eu> >> Cc: snmp4j@agentpp.org >> Subject: Re: [SNMP4J] max-bindings with big tables >> >> Hi Steffen, >> I will try to reproduce this issue. >> Independent from the result, the parameters for TableUtils are not suitable >> for your setup. The maxNumColumnsPerPDU has to be as large as possible. >> Otherwise the overall performance will be bad and the likelihood of >> incomplete table rows increases significantly (through changes in the agent >> while TableUtils operate). >> Best regards >> Frank >> >>> Am 06.07.2018 um 10:20 schrieb Steffen Brüntjen >>> <steffen.bruent...@macmon.eu>: >>> >>> Hi! >>> >>> I'm using SNMP4J version 2.6.2. >>> >>> Best regards >>> Steffen >>> >>> -----Original Message----- >>> From: Frank Fock [mailto:f...@agentpp.com] >>> Sent: Donnerstag, 5. Juli 2018 19:37 >>> To: Steffen Brüntjen <steffen.bruent...@macmon.eu> >>> Cc: snmp4j@agentpp.org >>> Subject: Re: [SNMP4J] max-bindings with big tables >>> >>> Hi Steffen >>> What SNMP4J version are you using? >>> Best regards >>> Frank >>> >>>> Am 05.07.2018 um 17:04 schrieb Steffen Brüntjen >>>> <steffen.bruent...@macmon.eu>: >>>> >>>> Hi Frank >>>> >>>> I believe I found an issue in the TableUtils class. In certain scenarios, >>>> the returned List<TableEvent> from getTable(Target target, OID[] >>>> columnOIDs, OID lowerBoundIndex, OID upperBoundIndex) will contain >>>> incomplete and duplicate rows. >>>> >>>> >>>> Here's an extract of an exemplary List<TableEvent> for a "good" result: >>>> >>>> [1.3.6.1.2.1.31.1.1.1.1.278 = VLAN105, [...], 1.3.6.1.2.1.31.1.1.1.18.278 >>>> = service] >>>> [1.3.6.1.2.1.31.1.1.1.1.279 = VLAN106, [...], 1.3.6.1.2.1.31.1.1.1.18.279 >>>> = reception] >>>> [1.3.6.1.2.1.31.1.1.1.1.283 = VLAN110, [...], 1.3.6.1.2.1.31.1.1.1.18.283 >>>> = voice] >>>> [1.3.6.1.2.1.31.1.1.1.1.373 = VLAN200, [...], 1.3.6.1.2.1.31.1.1.1.18.373 >>>> = clients] >>>> [1.3.6.1.2.1.31.1.1.1.1.774 = VLAN601, [...], 1.3.6.1.2.1.31.1.1.1.18.774 >>>> = VLAN601] >>>> [1.3.6.1.2.1.31.1.1.1.1.783 = VLAN610, [...], 1.3.6.1.2.1.31.1.1.1.18.783 >>>> = lab6] >>>> >>>> >>>> But in some specific circumstances, I get results like these: >>>> >>>> [ ... 75 normal rows ... ] >>>> [1.3.6.1.2.1.31.1.1.1.1.278 = VLAN105, [...], 1.3.6.1.2.1.31.1.1.1.18.278 >>>> = service] >>>> [1.3.6.1.2.1.31.1.1.1.1.279 = VLAN106, [...], 1.3.6.1.2.1.31.1.1.1.18.279 >>>> = reception] >>>> [null, null, null, null, 1.3.6.1.2.1.31.1.1.1.14.283 = 2, >>>> 1.3.6.1.2.1.31.1.1.1.15.283 = 0, 1.3.6.1.2.1.31.1.1.1.18.283 = voice] >>>> [null, null, null, null, 1.3.6.1.2.1.31.1.1.1.14.373 = 2, >>>> 1.3.6.1.2.1.31.1.1.1.15.373 = 0, 1.3.6.1.2.1.31.1.1.1.18.373 = clients] >>>> [null, null, null, null, 1.3.6.1.2.1.31.1.1.1.14.774 = 2, >>>> 1.3.6.1.2.1.31.1.1.1.15.774 = 0, 1.3.6.1.2.1.31.1.1.1.18.774 = VLAN601] >>>> [null, null, null, null, 1.3.6.1.2.1.31.1.1.1.14.783 = 2, >>>> 1.3.6.1.2.1.31.1.1.1.15.783 = 0, 1.3.6.1.2.1.31.1.1.1.18.783 = lab6] >>>> [1.3.6.1.2.1.31.1.1.1.1.283 = VLAN110, 1.3.6.1.2.1.31.1.1.1.17.283 = 2, >>>> 1.3.6.1.2.1.31.1.1.1.6.283 = 0, 1.3.6.1.2.1.31.1.1.1.10.283 = 0, null, >>>> null, null] >>>> [1.3.6.1.2.1.31.1.1.1.1.373 = VLAN200, 1.3.6.1.2.1.31.1.1.1.17.373 = 2, >>>> 1.3.6.1.2.1.31.1.1.1.6.373 = 0, 1.3.6.1.2.1.31.1.1.1.10.373 = 0, null, >>>> null, null] >>>> [1.3.6.1.2.1.31.1.1.1.1.774 = VLAN601, 1.3.6.1.2.1.31.1.1.1.17.774 = 2, >>>> 1.3.6.1.2.1.31.1.1.1.6.774 = 0, 1.3.6.1.2.1.31.1.1.1.10.774 = 0, null, >>>> null, null] >>>> [1.3.6.1.2.1.31.1.1.1.1.783 = VLAN610, 1.3.6.1.2.1.31.1.1.1.17.783 = 2, >>>> 1.3.6.1.2.1.31.1.1.1.6.783 = 0, 1.3.6.1.2.1.31.1.1.1.10.783 = 0, null, >>>> null, null] >>>> [ ... everything normal ... ] >>>> >>>> >>>> Here we find some rows split into two: One block with the first 4 columns >>>> set null, and another block with the last 3 columns set null. >>>> >>>> >>>> Here's the setting which produces the second result: >>>> >>>> - max-bindings is set to 4 - TableUtils.setMaxNumColumnsPerPDU(int) >>>> - max-repetitions is set to 30 - TableUtils.setMaxNumRowsPerPDU(int) >>>> - the device returns many rows (like 120) >>>> - the table request contains more columns than max-bindings >>>> - the table request contains not a multiple of max-bindings >>>> - the problem will also depend on MTU size, but that's not important here >>>> >>>> >>>> This is what happens: >>>> >>>> 1. TableUtils will request the first 4 columns >>>> 2. device returns 60 variable bindings, that's 15 cells per column >>>> 3. TableUtils will request the latter 3 columns >>>> 4. device returns 60 variable bindings, that's 20 cells per column >>>> >>>> This is repeating until all bindings are retrieved. So far, so good. The >>>> problem is now, that all second requests (step 3) will receive more rows, >>>> and so these requests will reach index 283 (as in the example above) >>>> earlier. I did some debugging and I think I found the reason: When the >>>> first results with index 283 are received (step 3), TableUtils creates a >>>> row for this index. That row is filled up with null values for the first 4 >>>> columns so that it's size equals 7 (and not 3). Having size=7, the row is >>>> considered finished too soon. TableUtils then prunes these incomplete but >>>> finished rows from rowCache. When TableUtils receives the other 4 columns >>>> for row 283, it creates a new row with the same index. >>>> >>>> >>>> How to fix? >>>> >>>> I believe a moderately easy, but not very good way to fix this is to have >>>> the little part contain the first 3 columns, not the remaining last 3 >>>> columns: >>>> >>>> max-bindings = 4 >>>> columns: .1, .2, .3, .4, .5, .6, .7 >>>> 1. packet should contain: .1, .2, and .3 >>>> 2. packet should contain: .4, .5, .6, and .7 >>>> >>>> Number of columns for the first packet is NumColumnsTotal % maxBindings. >>>> Number of columns for the other packets is maxBindings. >>>> >>>> >>>> Please tell me if you need more information or if my method invocation is >>>> wrong. >>>> >>>> >>>> Best regards >>>> Steffen Brüntjen >>>> _______________________________________________ >>>> SNMP4J mailing list >>>> SNMP4J@agentpp.org >>>> https://oosnmp.net/mailman/listinfo/snmp4j >>> >> > > _______________________________________________ > SNMP4J mailing list > SNMP4J@agentpp.org > https://oosnmp.net/mailman/listinfo/snmp4j _______________________________________________ SNMP4J mailing list SNMP4J@agentpp.org https://oosnmp.net/mailman/listinfo/snmp4j
