Re: discussion generic import design

[Christian Stimming]

Ok, I see we are in fact getting somewhere. Sorry that I didn't state my intentions earlier. I now see that one reason why I'm unhappy with the design you decided on, is that eventually in fact the HBCI code needs something else than the OFX and other code. In HBCI, I can *always* rely on several assumptions that *greatly* simplify or streamline the importing process:

* The originating account for a transaction is always known
* The transaction currency is always known
* The HBCI importer can always specify only one split of the transaction (i.e. the originating one) [1]
* Transaction matching always happens according to the memo/description/payee string
* No transactions with interdependencies are imported, i.e. the protocol importer doesn't have transactions with both splits already specified but only with one split as mentioned above.
* The transactions to be imported are already available in one specific list, i.e. the importing process has a well-defined beginning and end.

I thought (and think) that the qif-importer represents a GUI that works very well with these assumptions given. The reason why I raised all this fuzz is that I realized that you were heading into a direction of a much more general GUI, but as I just described, for HBCI this would be an over-complexification of the whole importing process.

Conclusion so far: I think I will give the qif-importer GUI a try, but not because I don't like your code but rather because we now eventually realized that the requirements are in fact quite different. Therefore the generic importer is fine as it is, and should be further optimized for a generic (probably mostly file-based) import -- and as Derek already pointed out, it would be a great basis for import from CSV or XML or a whole lot of other account manager's file formats. But since HBCI comes with the very rigid restrictions mentioned above, a much more restricted GUI might do a better job there. Nevertheless in one week from now I might still realize that deriving from the qif-importer ends up way more borken and complex than ever before... then I might have to repent, tear my clothes and put ash on my head...

Some comments on the technical details below.

[1] There's one exception: If a user transfers money between several of his own HBCI-enabled accounts, then the HBCI importer might realize this fact, and then it can specify both splits. However, this is really a very rare exception, and also it could just be found by the "normal" transaction matching, since that will most likely take the "destination account code" into account for matching.

Benoit wrote:
> 1-(Not directly related to this particular issue, but a prerequisite):
> Generate a preference page, in which we can check in what options we want
> enabled (Currently the options are ADD, REPLACE, IGNORE, RECONCILE). That
> page should also contain:
> - -ADD and RECONCILE match threshold (two integers)
> - -Wether by default transaction should be ADDed or RECONCILEd with c (cleared)
> or y (reconciled). Someone who reconciles with a paper statement at the end
> of the month probably wants c, while someone who uses the matcher to add all
> his transaction frequently probably wants y. We could also make ADD and
> RECONCILE independent.

I appreciate the proposed compromise, and at this point in time this would be something that everbody could live with. However, have you followed the "general preference" discussion during the last days? They've argued about whether user-editable preferences should be created in the first place or not, and that creation of preferences should be avoided if they just cover a not-really-completed design. IMHO These preferences would unfortunately fit into exactly that category: Two integers for the Match Threshold may be fine for you to fine-tune the behaviour, but the ordinary user would just have Zero knowledge about what that integer should be. Maybe the user would be far better off if he wouldn't even now that the threshold can be varied. Maybe the threshold should only be varied between three text choices, "rigid, loose, very loose" or similar. Maybe the generic importer can even put some more magic into that process and e.g. remember the user's choices as "average matching value that the user accepted", and use a function of that as match threshold. I'm just throwing ideas out here, these don't have to implemented -- I just want to underline that creating a user-visible preference here might actually not be a favor to the user.

> 3-The cell containing the action is text, so it isn't obvious at all that you
> can take an action there. A pixmap representing some sort of button must be
> created for each action. The design should ideally make it clear that this
> button iterates over a list of choices (which we unfortunately can't
> display). Basically this is a spin button that spins only one way.

I totally agree that a cell containing text isn't clear enough about that this can do some action. Obviously you know better than me about what is possible in the Clist. Creating button-looking pixmaps as a workaround might do, but: do these pixmaps contain text? If yes, the translators would need to translate the pixmaps as well, which is not so good. Maybe your current solution with a Box of Buttons next to the transaction list wasn't so bad after all. :-) Maybe it will be more intuitive if the buttons are somewhere grouped with the right split-list, since that too changes from transaction to transaction.

>>There's one more issue with the generic importer, namely, the imported
>>transactions are already created in the final accounts currently and they
>>are already visible in that account's registers.
>
> I agree seeing them in the register below is a nuisance.
> It a minor problem for OFX, but I think in your case you have the register
> open for the account being imported, so it must be a real nuisance.

Absolutely right. Sorry that I didn't state that earlier.

>>avoid that. Instead, I would suggest that the generic importer should
>>define its own "transaction" data type, which might be something like
>>
>>struct ImportTrans {
>> /* gnucash transaction with everything filled and with one split, but
>> which was not yet inserted into the account */
>> Transaction *trans;
>
> You can't, according to Transaction.h, a split MUST be parented by an account
> before you use any of the xaccSplitSetAmount() and similar functions on it.
> And since an imported transaction could have any number of splits, the splits
> must be set.

Oops, that's right. But: For HBCI, again I know that I always have only one split.

But in general: It would still be possible for the generic importer to not only define its own transaction data type but also its own split data type. This is precisely what the qif-importer does (with types defined in qif-objects.scm).

struct ImportTrans {
Transaction *trans;
GList *split_list;
char *payee_memo;
}
struct ImportSplit {
char *memo;
gnc_numeric amount;
Account *account;
}

or similar. Then the importer will create the "real" split for each ImportSplit, add it to the respective account, fill in the amount, add it to the transaction etc. etc.

As for the lookup table of payee/memo matching: The qif-importer implements it in a scheme "alist" which is stored from session to session in a seperate file in $HOME/.gnucash/qif-map-something . Whether a seperate file or some KVP would be the better place to store it --- I don't really care, since I think it doesn't really make much of a difference. So that's just an implementation question.

Christian

_______________________________________________
gnucash-devel mailing list
[EMAIL PROTECTED]
http://www.gnucash.org/cgi-bin/mailman/listinfo/gnucash-devel