In fact, running the same thing on a mac gives 10-40% speedup (first
benches were on linux javx2):
# dat1 =: 1!:1 < 'data/csv1.csv'
62085027
# dat2 =: 1!:1 < 'data/csv2.csv'
91614581
# dat3 =: 1!:1 < 'data/csv3.csv'
236024379
cmd =: > 'pdsv '&, &.> ;: 'dat1 dat2 dat3'
load 'tables/dsv'
pdsv =: (',';'"')&fixdsv
]cur =: 6!:2"1 cmd
3.5321 4.18567 19.5861
load '../tables_dsv/dsv.ijs'
pdsv =: (',';'"')&fixdsv
]df0 =: 6!:2"1 cmd
2.2922 3.68782 16.3072
load '../base9/main/strings.ijs'
pdsv =: (',';'"')&fixdsv
]df1 =: 6!:2"1 cmd
2.13638 3.32746 12.8913
cur (- % [) df0
0.351038 0.118942 0.167412
cur (- % [) df1
0.395152 0.205036 0.341815
On 11/10/20, Joseph Novakovich <[email protected]> wrote:
> I copy/pasted the benchmark code if you follow the link to tables_dsv:
>
> https://github.com/jsoftware/tables_dsv/pull/1
>
> Basically, it's just 6!:2 on a few csvs of size ~50-250 MB.
>
> On 11/10/20, Raul Miller <[email protected]> wrote:
>> 10-20% isn't much -- differences in data (for example, because of
>> size) may easily be more significant, and different platforms may also
>> change things around.
>>
>> Which brings up the question: what did your testing look like?
>>
>> Thanks,
>>
>> --
>> Raul
>>
>> On Tue, Nov 10, 2020 at 1:25 PM Joseph Novakovich
>> <[email protected]> wrote:
>>>
>>> Hello,
>>>
>>> I noticed a spot for performance improvement in 'fixdsv' and
>>> 'chopstring'. I opened two pull requests on github for the base9
>>> library and the tables/dsv addon:
>>>
>>> https://github.com/jsoftware/tables_dsv/pull/1
>>> https://github.com/jsoftware/base9/pull/1
>>>
>>> Basically, bitmasks were using 'e.' to find occurrences of a single
>>> char in the input. Using '=' instead seems to give around 10-20%
>>> speedup.
>>>
>>> Joseph
>>> ----------------------------------------------------------------------
>>> For information about J forums see http://www.jsoftware.com/forums.htm
>> ----------------------------------------------------------------------
>> For information about J forums see http://www.jsoftware.com/forums.htm
>>
>
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
