On Fri, Jun 14, 2019 at 3:36 PM Christopher Singley <[email protected]> wrote:
> > > On Wednesday, 12 June 2019 22:31:17 UTC-5, Martin Blais wrote: >> >> Thanks for your interest Christopher. >> > > Well it is interesting, to me at least. I enjoy looking through your > code; I'm well familiar with the problem space, but your methods are quite > different. > > >> The circular nature of the problem is that interpolation may depend on >> booking, and booking may depend on interpolation. >> There's no perfect solution to that, that I could find.ave >> > > You mean "booking" in the sense of "realizing capital gains", yes? Why > does booking depend on interpolation? Is this because of your emphasis of > specific identification as a cost accounting method? There are ways out of > that. > No; it would be very easy to fix if the issue was just implementing different booking methods :-) The problem occurs because the syntax I created specifically aims to allow users to elide some information and automatically fill in some missing numbers. For instance, you don't have to provide all the details of a reducing lot, as long as the list of lots it matches (when filtered down) yields an unambiguous set (either a single lot, or many lots for which the total number of units matches the size of the reducing lot precisely). This is the process I call "booking", that is, matching a partial specification for reducing lots against the available lots just before the transaction gets applied. It uses the accumulated inventory in order to fill in missing information. "Interpolation," on the other hand, is a similar process that fills in missing numbers, but not by matching against the contents of the inventory just before the transaction gets applied, but rather only against the other postings, by assuming that the set of postings for each currency group must balance. This does not make use of the state of the inventory before the transaction gets applied, just the information provided on that one transaction. It basically attempts to figured out the cost currency of each posting, then groups them by cost currency, and then attempts to fill in missing bits and piece (either numbers or currencies) in each of these currency groups. These two are similar in goal: fill in missing information automatically to ease the burden of data entry, but in some cases - cases which are which particular bits are left missing in the input and for Beancount to figure out - running booking before interpolation works, and in other cases running interpolation preceding booking works. I have seen cases that are impossible to resolve. It took me a while to figure out which order was the most useful in practice, and this is what's in there now. Specific identification is a very uncommon cost accounting method. It's > almost always FIFO or (for mutual fund companies) the degenerate average > cost method. It's good to support specific identification (generality is > good!) but given its rarity, it's not unreasonable to enforce a requirement > that opening/closing transactions (or augmenting/reducing transactions in > your usage) must have matching labels if you want to use specific > identification. Don't attempt to interpolate the opening transaction from > date/price, and the problem is solved, no? > > Is guessing the opening transaction from partial user input (i.e. date or > price) a high priority? The algorithm cannot reliably find a solution > because of underspecified inputs, as you note in your docs, and it requires > the user to manually duplicate a significant effort by keeping their own > inventory outside of beancount (probably in a spreadsheet). I don't know > about you, but never maintaining another lot-matching spreadsheet ever > again is very high on my list of priorities. > > You already have a good chunk of the inventory system built into > beancount. If you let go of the requirements that are introducing > recursion into your algorithm, I guess you'd find the benefits a lot more > valuable than the bits of interpolation that you'd need to drop in order to > achieve it. > > I've written an inventory system that does this for me, so I know it can > be done. I doubt you'd find it terribly useful, but if you're interested I > can show you how I handle cost accounting. I've got a Python package that > handles trades, splits, spinoffs, mergers, return of capital distributions, > all that fun stuff. It's somewhat battle-tested, too, with a relatively > high volume of messy real-world transactions run through it, and the > results audited (as in CPAs engaged to discover discrepancies, not just > unit tests). You might find it interesting to look at an alternative > implementation. The code won't win any prizes for engineering elegance, > and still needs some work, but the output is demonstrably correct for the > most part. > I'd be curious to have a look, but unfortunately I'm too busy right now, I have very little time, just keeping my head above water, mostly. > > > The implementation of the Inventory has already evolved since this was >> written (for performance reasons) and IIRC is treated mostly like a list, >> matching portions that have been specified to filter a list candidate >> positions. I'm not beyond reviewing core classes - especially if it might >> help - but I believe changing the mapping would make no difference at all >> here. I wrote an example some time ago - in a text file IIRC, which I >> shared on the list and had some comments about - but I can't seem to find >> it right now. >> > > I'd be interested in seeing the doc if you happen to stumble across it, > but it's not a big deal. You're right that the dict keys aren't a deal > breaker; they can be worked around without much trouble. I'm still > puzzling through how you do this. > I'll bring it up if I can find it. It was a text file in another branch IIRC, in the midst of code. > > Thanks for releasing beancount, it's nice software > Thank you! On Wed, Jun 12, 2019 at 12:27 AM Christopher Singley <[email protected]> >> wrote: >> >>> I've been reading through this: >>> >>> http://furius.ca/beancount/doc/self-reductions >>> >>> and puzzling through parser.booking_full. >>> >>> It looks to me like the root cause of your struggles is that the keys to >>> your >>> Inventory mapping are overspecified - the necessity to perform the >>> calculations to >>> populate a Cost instance in order to look up a lot. I reckon you need >>> to move >>> the cost data from keys to values, so that inventory is a mapping from >>> (account, security) -> [(units, cost, date)] >>> instead of the current mapping from >>> (account, security, cost, date) -> [(units, )]. >>> The former is a more natural data structure for cost accounting. >>> >>> Any well-formed transaction natively has (account, security) fields. >>> Use those to look up a sequence of lots containing (lot_units, cost, >>> open_date). >>> Filter that sequence using (transaction_date, transaction_units) to find >>> lots that >>> might be closed ("booked") by the incoming transaction - it will >>> definitely have >>> transaction_date, and if it doesn't have transaction_units for some >>> reason, then that >>> is trivially interpolated. >>> >>> Next step depends on your cost accounting method. Normally you'd sort >>> transactions >>> by date/time to do FIFO or average cost. To instead do specific >>> identification, you'd >>> further filter the lots for a particular date/cost/label, and require a >>> unique result. >>> >>> NOW you do the heavy lifting. >>> >>> Work through the surviving lots in order, popping lots and splitting >>> them as necessary >>> until you run out of transaction_units or lot_units. For each popped >>> lot, couple its >>> data to (transaction_date, transaction_units, transaction_price), >>> and you'll have all the data needed to fully populate a journal entry. >>> >>> There's nothing recursive about this calculation. You can implement it >>> as a straight >>> pipeline of iterators, evaluated lazily. >>> >>> An additional advantage is that this procedure is easy to extend to >>> handling other >>> securities transaction types that don't involve realizing gain. >>> E.g. for a split, use (account, security) to look up your position. >>> Filter that sequence for lots with an open_date before the >>> transaction_date, >>> and replace them with copies with the units/cost adjusted for the split. >>> Keep a running total of the change in units, and require that total to >>> match the input >>> transaction_units (which is a hard requirement for a stock split >>> transaction). >>> >>> Any conceptual problems with this setup? I mean, other than being a >>> huge PITA to >>> rip up existing classes and everything that touches them. >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Beancount" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/beancount/e84f305c-9282-1d20-d74d-00d99620f2da%40singleys.com >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- > You received this message because you are subscribed to the Google Groups > "Beancount" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/beancount/66479691-deff-4982-994f-845d71aae0e1%40googlegroups.com > <https://groups.google.com/d/msgid/beancount/66479691-deff-4982-994f-845d71aae0e1%40googlegroups.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Beancount" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/beancount/CAK21%2BhMXumgJXtrBXV2B7Edu%2BQTK5yygwmtKGcOt4bt%3Dcmrkgw%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
