Beyond dictionaries is a "generic data structure" construct, and a large
extended family of data structures that can respond to the following commands
GET SET UPSERT APPEND DELETE INSERTafter(item) INSERTbefore(item) WHEREget
CONVERTtoJnativearray UPSERTfromJnativearray
in addition to dictionaries, ragged nested arrays or generally any binary
encoding of datatype;shape/count;content
for dictionaries, some applications (or at least language implementations)
permit variant/inconsistent type keys and/or values.
and generally, a table is a dictionary with
multiple keys and multiple associated values to each unique combination of the
multiple keys. With potential performance advantages to keeping it in a sorted
order if retrieval of items by query, or having multiple fields indexed. Typed
"tables"/dictionaries get inherent performance advantages due to not requiring
item boxing.
as a generic "pattern framework" for data structures without using classes:
DEFAULT =: ''
data dataconstructor DEFAULT NB. dataconstructor is conjunction that will
return an adverb that is (conjunction data). example
mydict = ('key1'; value1) dict DEFAULT NB. an adverb that is a conjunction
bound to specific data, and where the conjunction code responds to commands in
list above top.
The list of commands near top of this message are actually numeric codes that
get passed to the bound datastructure adverb.
GET mydict becomes a verb that will retrieve items.
SET mydict returns an conjunction with m as key, n as value that will return
the same mydict adverb but with updated "data"
(newkey; SET) mydict returns an adverb that is above conjunction with m bound
as newkey
(newkey: newdata; SET) mydict directly returns the updated mydict adverb.
In the case of typed/consistent dictionaries, the internal data representation
is just an inverted table of "fields". And while the underlying conjunction of
mydict is bound to data to form an adverb, a generic (consistent typed)
dictaccess conjunction can operate on any inverted table if prebound with
GET/SET. And the GET/SET/OTHER "commands" can form a verb as a result of
paired with a generic access adverb, all "command parameters" as x, and the
inverted table/data structure as y.
What makes this approach have more appeal to be over classes is besides issues
like reference management and it is hard to send methods to class/objects in a
"parameterized" way, is that arrays are J's first/primary datastructure, and in
some datastructures can be transparently accessed by independent J code.
Generally, putting the complexity in modifiers instead of classes is a
functional approach I would prefer. I can volunteer to start it off at least
if there is interest in this approach.
It would be a datastructure access "language/dsl" on top of J.
On Monday, January 31, 2022, 01:43:30 p.m. EST, Henry Rich
<[email protected]> wrote:
Quite right. Jan-Pieter has at least a good start on the model. What I
am suggesting is that the 'more integrated implementation' might not
need to be much more integrated than the J script. Searching and
sorting are already J primitives.
The deficiency in J is that m&i. gives you a hashtable for searching m,
but if you modify m you have to recalculate the hash from scratch. This
makes m&i. a good read-mostly Dictionary but slow for frequent updates.
If we can refine the model to the point that such inefficiency is the
worst problem we have, a little integrated support could finish the package.
Henry Rich
On 1/31/2022 1:37 PM, Eric Iverson wrote:
> A good and complete model is the first step. Then it is a matter of how
> much it is used and what drawbacks it might have that would be addressed by
> a more integrated implementation.
>
> On Mon, Jan 31, 2022 at 1:27 PM Henry Rich <[email protected]> wrote:
>
>> I have looked into this quite a bit. I am not convinced that Dictionary
>> is a fundamental datatype like number or letter, or that the current
>> symbol support is deficient. That makes the first questions What is a
>> Dictionary? and Where can a Dictionary be used in J?
>>
>> The use case that would be important to me is a key-value store, aka
>> associative memory, where the Dictionary remember keys and values and
>> later returns a value when presented with the key. This feels to me like
>> a package written in J rather than a J primitive. A Dictionary would be
>> a numbered locale. The place where I could see added support inside J
>> would be in adding to and deleting from hashtables, specifically ones
>> created/used by m&i. .
>>
>> I invite proposals on what a Dictionary package needs to do. I have
>> done this before, and Jan-Pieter Jacobs responded with a package.
>> Quoting from his message of 20210327:
>>
>> *** QUOTE
>>
>> install'github:jpjacobs/types_dict@main'
>>
>> (except on (my) android, where github installs don't seem to work, neither
>> in the Gui version JA nor on the commandline via termux)
>>
>> It is pretty simple in use, you just use:
>> d=: dict keys;vals
>> for creating the dictionary, which is a OOP object, having the following
>> methods:
>> get, set, map, sort, destroy, whose documentation is contained in
>> help_pdict_
>>
>> For performance, the create verb precomputes the lookup for the get verb,
>> both forward get (key->value) and backward get inv (value->first key) upon
>> object creation.
>>
>> *** END QUOTE
>>
>> I have not looked into this package, but it seems to me to have the
>> right entry points. Can we take that as a starting point to see what
>> needs to be added? The first step of course would be to put the addon
>> under Package Manager.
>>
>> Henry Rich
>>
>>
>>
>> On 1/31/2022 12:58 PM, Elijah Stone wrote:
>>> I agree with Eric regarding the challenges of adding dictionaries.
>>>
>>> One issue: I think a necessary prerequisite is improved symbols.
>>>
>>> On Mon, 31 Jan 2022, Alex Shroyer wrote:
>>>
>>>> I agree with Raoul that competing with Python is not a good idea.
>>>> But J can learn from Python's decisions (good and bad) to grow and
>>>> improve.
>>>> In my opinion, numpy is Python's "killer app" because it brings
>>>> reasonable
>>>> performance without much conceptual overhead.
>>>> The feature of Python that enabled numpy is its extensibility, down
>>>> to the
>>>> C layer.
>>>>
>>>> There are other good features of Python that J could copy, in particular
>>>> the 'dictionary' data type.
>>>> The array language K has dictionaries, so J might take some inspiration
>>>> from there for integrating them into the language.
>>>>
>>>> Cheers,
>>>> Alex
>>>>
>>>>
>>>> On Mon, Jan 31, 2022 at 5:30 AM Raoul Schorer <[email protected]>
>>>> wrote:
>>>>
>>>>> Just my 2c, but I think that competing with python in general is
>>>>> somewhat
>>>>> delusional. I think the key point for expanding J use to have a
>>>>> "killer J
>>>>> app". For example, an improved clone of or excellent plugin for
>>>>> VisiData (
>>>>> https://www.visidata.org/) is my idea of a killer app. But someone
>> here
>>>>> may
>>>>> have a better idea?
>>>>>
>>>>> Cheers,
>>>>> Raoul
>>>>>
>>>>> Le lun. 31 janv. 2022 à 03:49, Ric Sherlock <[email protected]> a
>>>>> écrit :
>>>>>
>>>>>> Yes from a data structure point of view, inverted tables get you a
>>>>> lot of
>>>>>> the way (note they're also available in the 'general/misc' addon -
>>>>> load
>>>>>> 'general/misc/inverted' ) and I've used them to good effect in my
>>>>>> 'data/struct' addon (https://github.com/tikkanz/data_struct).
>>>>>> I agree that J's arrays are more general, flexible & powerful. But
>>>>> when
>>>>>> you're dealing with a tabular data set there is an overhead to
>>>>> keeping
>>>>> the
>>>>>> fields in a table in sync that dataframes can help with. Perhaps it's
>>>>>> something abstracting the structure so you don't have to deal so much
>>>>> with
>>>>>> the mechanics of manipulating it? At least for me :)
>>>>>>
>>>>>> On Mon, Jan 31, 2022 at 3:28 PM Elijah Stone <[email protected]>
>>>>> wrote:
>>>>>>> https://code.jsoftware.com/wiki/Essays/Inverted_Table, perhaps?
>>>>>>>
>>>>>>> That said, I think a great strength of j is that data are _not_
>>>>>> explicitly
>>>>>>> tabular. The associations are defined in an ad-hoc manner as
>>>>> needed by
>>>>>>> the programmer. This is also an essential difference between
>>>>> the array
>>>>>>> paradigm and the relational paradigm (cf SQL): in the former,
>>>>> pointers
>>>>>>> come with implicit context; in the latter, that context must be
>>>>> explicit.
>>>>>>> -E
>>>>>>>
>>>>>>> P.S. regarding analysis/optimization: I would love to see it,
>>>>> but for
>>>>>> some
>>>>>>> reason everybody is scared of building a compiler because of the
>>>>> parsing
>>>>>>> problem.
>>>>>>>
>>>>>>> On Mon, 31 Jan 2022, Ric Sherlock wrote:
>>>>>>>
>>>>>>>> Yes, I've been thinking that a Dataframes equivalent in J
>>>>> would be
>>>>>>> useful.
>>>>>>>> Most things are already possible with J's arrays, but
>>>>> conceptually
>>>>>>>> DataFrames are well understood by many now, and they make it
>>>>> easy to
>>>>>> work
>>>>>>>> with datasets as named fields.
>>>>>>>> I've spent a reasonable amount of time working with Pandas,
>>>>> but have
>>>>>>>> recently been using Polars (Rust backend with Python bindings)
>>>>> which
>>>>>>> really
>>>>>>>> shines for larger datasets. Performance (especially
>>>>> read/write) is
>>>>>>> awesome,
>>>>>>>> and the LazyFrames which optimise your query/analysis plan
>>>>> make a big
>>>>>>>> difference too.
>>>>>>>> I haven't taken enough time to explore it, but maybe Jd is the
>>>>> starting
>>>>>>>> point in this space for J?
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jan 31, 2022 at 9:01 AM Michail L. Liarmakopoulos <
>>>>>>>> [email protected]> wrote:
>>>>>>>>
>>>>>>>>> Hello all,
>>>>>>>>>
>>>>>>>>> I find any parallels between python and J pretty interesting,
>>>>> being
>>>>> a
>>>>>>>>> person with some python experience and an interest of the
>>>>> applications
>>>>>>> of
>>>>>>>>> both python and J in mathematical modelling, analytics,
>>>>> computational
>>>>>>> math
>>>>>>>>> and perhaps computational physics too.
>>>>>>>>>
>>>>>>>>> If you'd like to bring some features from the python
>>>>> math/analytics
>>>>>>>>> libraries/ecosystem in J, I'd suggest you to look at the
>>>>> features of
>>>>>>> three
>>>>>>>>> libraries:
>>>>>>>>>
>>>>>>>>> - numpy (I believe most features are already covered from the
>>>>> built
>>>>> in
>>>>>>>>> features of an array language such as J)
>>>>>>>>>
>>>>>>>>> - pandas ( a nice library for manipulating csv files within
>>>>> python
>>>>> as
>>>>>>>>> dataframe objects -- see the dataframes from the R language)
>>>>>>>>>
>>>>>>>>> - scipy (a collection of methods and functions ranging from
>>>>> solving
>>>>>>>>> numerically: differential equations, evaluating definite
>>>>> integrals,
>>>>>>>>> constrained and unconstrained optimization, and I believe
>>>>> statistics
>>>>>>> too)
>>>>>>>>> There is also out there an amazing python library for symbolic
>>>>>>> calculations
>>>>>>>>> (like the ones you can do with Mathematica or WolframAlpha:
>>>>> symbolic
>>>>>>>>> evaluation of definite and indefinite integrals, symbolic
>>>>> solutions
>>>>> of
>>>>>>>>> diff. equations, symbolic solutions of algebraic and Diophantine
>>>>>>> equations
>>>>>>>>> etc...). It's called sympy.
>>>>>>>>>
>>>>>>>>> But I'm not sure if you'd like J to include symbolic
>>>>> computations
>>>>> too
>>>>>>> or if
>>>>>>>>> the aim of the language is to excel only in numerics, data
>>>>> analytics,
>>>>>>>>> stats, whatever can be quantified pretty much.
>>>>>>>>>
>>>>>>>>> My few cents as food for thought.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>>
>>>>>>>>> ---
>>>>>>>>> Michail L. Liarmakopoulos, MSc
>>>>>>>>>
>>>>>>>>> On Sun, Jan 30, 2022, 20:39 R.E. Boss <[email protected]>
>>>>> wrote:
>>>>>>>>>>
>>>>>>>>>> I copied the first chapter of the book A Journey to Core
>>>>> Python
>>>>> (in
>> https://drive.google.com/file/d/1p1uIANh-LFniNNRqjDeeWWd4_-ddEZmz/view?usp=sharing
>>>>>>>>> )
>>>>>>>>>> and have the question: do we want that J is competitive with
>>>>> Python?
>>>>>>>>>> If the answer is yes, the next question is: what is the to
>>>>> do list
>>>>>> to
>>>>>>> be
>>>>>>>>>> competitive and how long will it take?
>>>>>>>>>>
>>>>>>>>>> (And then the unavoidable question: WHY?)
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Personally I think we must aim on the niches in the market, as
>>>>> there
>>>>>>> are
>>>>>>>>>> the mathematical oriented people, e.g. the broad scientific
>>>>>> community.
>>>>>>>>>> Then all people experienced in Excel or other spreadsheets or
>>>>>>> calculation
>>>>>>>>>> tools.
>>>>>>>>>>
>>>>>>>>>> Schools and universities. Financial and statistical oriented
>>>>> people.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> What we should do, IMHO, is
>>>>>>>>>>
>>>>>>>>>> - to emphasize the strengths of J,
>>>>>>>>>>
>>>>>>>>>> - to improve (considerably) the error handling of J,
>>>>>>>>>>
>>>>>>>>>> - to admit the steep learning curve,
>>>>>>>>>>
>>>>>>>>>> - to facilitate the use of mnemonics instead of primitives (I
>>>>> tried
>>>>>>> this
>>>>>>>>>> afternoon the primitives.ijs for half an hour, but was not
>>>>> capable
>>>>>> of
>>>>>>> use
>>>>>>>>>> any mnemonic, not even with
>>>>>>>>>> https://code.jsoftware.com/wiki/Primitives_to_Mnemonics)
>>>>>>>>>>
>>>>>>>>>> - to decide which of the features, benefits or applications
>>>>> (of
>>>>>>> Python)
>>>>>>>>> we
>>>>>>>>>> want J to have.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Just my 2 cents.
>>>>>>>>>>
>>>>>>>>>> R.E. Boss
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>> ----------------------------------------------------------------------
>>>>>>>>>> For information about J forums see
>>>>>>> http://www.jsoftware.com/forums.htm
>>>>> ----------------------------------------------------------------------
>>>>>>>>> For information about J forums see
>>>>>> http://www.jsoftware.com/forums.htm
>>>>> ----------------------------------------------------------------------
>>>>>>>> For information about J forums see
>>>>> http://www.jsoftware.com/forums.htm
>>>>> ----------------------------------------------------------------------
>>>>>>> For information about J forums see
>>>>> http://www.jsoftware.com/forums.htm
>>>>> ----------------------------------------------------------------------
>>>>>> For information about J forums see
>>>>> http://www.jsoftware.com/forums.htm
>>>>> ----------------------------------------------------------------------
>>>>> For information about J forums see http://www.jsoftware.com/forums.htm
>>>>>
>>>> ----------------------------------------------------------------------
>>>> For information about J forums see http://www.jsoftware.com/forums.htm
>>> ----------------------------------------------------------------------
>>> For information about J forums see http://www.jsoftware.com/forums.htm
>>
>> --
>> This email has been checked for viruses by AVG.
>> https://www.avg.com
>>
>> ----------------------------------------------------------------------
>> For information about J forums see http://www.jsoftware.com/forums.htm
>>
> ----------------------------------------------------------------------
> For information about J forums see http://www.jsoftware.com/forums.htm
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
----------------------------------------------------------------------
For information about J forums see http://www.jsoftware.com/forums.htm
