Re: Search Values of Array in "One Go"

[Bob Sneidar via use-livecode]

+5

Bob S


> On Aug 26, 2017, at 16:04 , Mark Wieder via use-livecode 
>  wrote:
> 
> filter lines of tList WHERE item 1 of each >= 1 into someVariable
> and
> filter elements of tArray where "late fee" is in each into newArray
> 
> -- 
> Mark Wieder


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Sannyasin Brahmanathaswami via use-livecode]

Thanks Brian…

We use MS office for enterprise now, So I have one drive and it came right thru.




 

On 8/26/17, 5:51 PM, "use-livecode on behalf of Brian Milby via use-livecode" 
 wrote:

Here is a link to the file on OneDrive:
https://1drv.ms/u/s!Apo0Atd869AFkjiW39Xit7Rmmp4b

(https***1drv.ms/u/s!Apo0Atd869AFkjiW39Xit7Rmmp4b)

If that won't work, I can upload to the forum somewhere.



___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Brian Milby via use-livecode]

Here is a link to the file on OneDrive:
https://1drv.ms/u/s!Apo0Atd869AFkjiW39Xit7Rmmp4b

(https***1drv.ms/u/s!Apo0Atd869AFkjiW39Xit7Rmmp4b)

If that won't work, I can upload to the forum somewhere.

On Sat, Aug 26, 2017 at 10:30 PM, Sannyasin Brahmanathaswami via
use-livecode  wrote:

> Please do post your stack… over at the forums?
>
> On 8/26/17, 12:21 PM, "use-livecode on behalf of Brian Milby via
> use-livecode"  use-livecode@lists.runrev.com> wrote:
>
> I'm using example 5 from this site for my JSON:
> https://adobe.github.io/Spry/samples/data_region/
> JSONDataSetSample.html
>
> I can post a copy of the test stack I'm playing around with if anyone
> would
> be interested.
>
>
>
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Sannyasin Brahmanathaswami via use-livecode]

Please do post your stack… over at the forums? 

On 8/26/17, 12:21 PM, "use-livecode on behalf of Brian Milby via use-livecode" 
 wrote:

I'm using example 5 from this site for my JSON:
https://adobe.github.io/Spry/samples/data_region/JSONDataSetSample.html

I can post a copy of the test stack I'm playing around with if anyone would
be interested.



___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Monte Goulding via use-livecode]

> On 27 Aug 2017, at 10:24 am, Mark Wieder via use-livecode 
>  wrote:
> 
> Well, yes and no.
> I just wanted to make sure that the 'where' clause doesn't mean that we also 
> can't have the 'into' clause.

Oh, sure I would expect if implemented that it wouldn’t inhibit the `into` 
clause.

Cheers

Monte
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Mark Wieder via use-livecode]

On 08/26/2017 04:11 PM, Monte Goulding via use-livecode wrote:



On 27 Aug 2017, at 9:04 am, Mark Wieder via use-livecode 
 wrote:

I assume that syntax would do what the filter command normally does in terms of 
modifying the contents of tList. I'd also like to see the form that returns the 
result rather than modifying the original, as in

filter lines of tList WHERE item 1 of each >= 1 into someVariable
and
filter elements of tArray where "late fee" is in each into newArray


We have that already ;-)


Well, yes and no.
I just wanted to make sure that the 'where' clause doesn't mean that we 
also can't have the 'into' clause.


--
 Mark Wieder
 ahsoftw...@gmail.com

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Monte Goulding via use-livecode]

> On 27 Aug 2017, at 9:04 am, Mark Wieder via use-livecode 
>  wrote:
> 
> I assume that syntax would do what the filter command normally does in terms 
> of modifying the contents of tList. I'd also like to see the form that 
> returns the result rather than modifying the original, as in
> 
> filter lines of tList WHERE item 1 of each >= 1 into someVariable
> and
> filter elements of tArray where "late fee" is in each into newArray

We have that already ;-)

Cheers

Monte
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Mark Wieder via use-livecode]

On 08/26/2017 02:43 PM, Monte Goulding via use-livecode wrote:



On 27 Aug 2017, at 7:40 am, Phil Davis via use-livecode 
 wrote:

Another one I would like:

   filter lines of tList WHERE item 1 of each >= 10


Yes I like that too… a tad more than `by`. It seems to read more naturally.


Since we're blue-skying this process anyway...

I assume that syntax would do what the filter command normally does in 
terms of modifying the contents of tList. I'd also like to see the form 
that returns the result rather than modifying the original, as in


filter lines of tList WHERE item 1 of each >= 1 into someVariable
and
filter elements of tArray where "late fee" is in each into newArray

--
 Mark Wieder
 ahsoftw...@gmail.com

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Brian Milby via use-livecode]

You could set your needle to be a valid LCS expression.  I just ran a test
using the following:

*do* "if (" & pNeedle & ") then put tPath into xFoundPaths[the number of
elements in xFoundPaths + 1]"

It took around 4x of the time to run though.  I set my needle to "tElement
> 5000" for one test and "tElement > 5000 and tElement < 5002" for
another.  Times were similar for each.

With a roughly 10% cost per nested if inside the recursion loop, it would
seem that specific helper functions would be needed for each type of
search.  I'm not sure if a switch would optimize it any.  The wrapper
function could be generalized and perform the if/switch once to select the
appropriate private handler.

Using matchtext for a simple "or" type doesn't seem to cost much more than
using a nested if would.  I tried a search using a needle of
"(Chocolate|Maple)".

I'm using example 5 from this site for my JSON:
https://adobe.github.io/Spry/samples/data_region/JSONDataSetSample.html

I can post a copy of the test stack I'm playing around with if anyone would
be interested.

On Sat, Aug 26, 2017 at 9:04 AM, Alex Tweedly via use-livecode <
use-livecode@lists.runrev.com> wrote:

> I think regex (or multiple string comparisons) is too limited a way to go.
>
> I would want to be able to do numeric comparisons, range comparisons, etc.
>
> e.g. x is a number, x<25, 24 < x < 33, etc.
>
> -- Alex.
>
> On 26/08/2017 05:05, Brian Milby via use-livecode wrote:
>
>> tElement isn't the part that would change (you are still searching against
>> a single value in the array), you would need to modify the comparison to
>> allow for pNeedle to contain multiple values and the operators.  It would
>> probably be easier to modify to use a RegEx.  If you have 2 comma
>> separated
>> items, you could use something like the following (I'm sure the more
>> experienced can offer much better ways though):
>>
>> *if* (tElement contains item 1 of pNeedle) or (tElement contains item 2 of
>> pNeedle) *then*
>>
>>
>> On Fri, Aug 25, 2017 at 10:41 PM, Sannyasin Brahmanathaswami via
>> use-livecode  wrote:
>>
>> OK I will bite
>>>
>>> what would an array representing an "AND/OR"
>>> style query look like such that if used as a predicate
>>>
>>> for
>>>
>>>   else if tElement contains pNeedle then
>>>
>>> it would return true/false
>>>
>>> How would you have to construct "tElement"
>>>
>>> ??
>>>
>>>
>>>
>>> Mark: In terms of generalizing that function - then the key line which
>>> checks
>>> whether an element matches is:
>>>
>>>   else if tElement contains pNeedle then
>>>
>>>
>>>  One could imagine that this could be an arbitrary predicate -
>>> pNeedle
>>>  doesn't have to be a string, so could be an array representing a
>>> AND /
>>>  OR style query; or could be a regex (in which case you'd use
>>> matchText).
>>>
>>> ___
>>> use-livecode mailing list
>>> use-livecode@lists.runrev.com
>>> Please visit this url to subscribe, unsubscribe and manage your
>>> subscription preferences:
>>> http://lists.runrev.com/mailman/listinfo/use-livecode
>>>
>>> ___
>> use-livecode mailing list
>> use-livecode@lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your
>> subscription preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
>>
>
>
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Monte Goulding via use-livecode]

> On 27 Aug 2017, at 7:40 am, Phil Davis via use-livecode 
>  wrote:
> 
> Another one I would like:
> 
>   filter lines of tList WHERE item 1 of each >= 10

Yes I like that too… a tad more than `by`. It seems to read more naturally.

Cheers

Monte
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Phil Davis via use-livecode]

On 8/26/17 2:23 PM, Monte Goulding via use-livecode wrote:

On 27 Aug 2017, at 5:57 am, J. Landman Gay via use-livecode 
 wrote:

This would be really useful. I'd suggest:

filter elements of tArray by tPath[each] is “baz”

Array paths don’t work like that. `each` here is the element and tPath is a 
path within it so it needs to be `each[tPath]` but I like `by`. Infact I 
couldn’t recall the syntax I liked for this when posting the email yesterday 
but now I remember it was `by` as it’s the same as sort.

A simpler example is:

filter lines of tList by item 1 of each >= 10


Another one I would like:

   filter lines of tList WHERE item 1 of each >= 10

Phil Davis



Cheers

Monte
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode


--
Phil Davis

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Monte Goulding via use-livecode]

> On 27 Aug 2017, at 5:57 am, J. Landman Gay via use-livecode 
>  wrote:
> 
> This would be really useful. I'd suggest:
> 
> filter elements of tArray by tPath[each] is “baz”

Array paths don’t work like that. `each` here is the element and tPath is a 
path within it so it needs to be `each[tPath]` but I like `by`. Infact I 
couldn’t recall the syntax I liked for this when posting the email yesterday 
but now I remember it was `by` as it’s the same as sort.

A simpler example is:

filter lines of tList by item 1 of each >= 10

Cheers

Monte
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[J. Landman Gay via use-livecode]

This would be really useful. I'd suggest:

filter elements of tArray by tPath[each] is “baz”

--
Jacqueline Landman Gay | jac...@hyperactivesw.com
HyperActive Software   | http://www.hyperactivesw.com



On August 26, 2017 1:14:59 AM Monte Goulding via use-livecode 
 wrote:


As a slightly related aside in this conversation I’d still one day like to 
add `each` to filter:


— filter array and retain elements where element[“foo”][“bar”] = “baz"
put “foo” into tPath[1]
put “bar” into tPath[2]
filter elements of tArray by evaluating each[tPath] is “baz”

Cheers

Monte
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your 
subscription preferences:

http://lists.runrev.com/mailman/listinfo/use-livecode




___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Richard Gaskin via use-livecode]

Alex Tweedly wrote:

> I think regex (or multiple string comparisons) is too limited a way to
> go.
>
> I would want to be able to do numeric comparisons, range comparisons,
> etc.
>
> e.g. x is a number, x<25, 24 < x < 33, etc.

The problem of querying hierarchically-ordered data has been around for 
a long time, and XQuery is a popular solution (at least when that data 
is XML).


Of course associative arrays are simpler than XML, so any XQuery-like 
solution crafted for LC could be even simpler to use.


But XQuery is quite powerful, probably a good model to consider for 
handling the variety of use-cases we'd want such a generalized query 
system to handle.


--
 Richard Gaskin
 Fourth World Systems
 Software Design and Development for the Desktop, Mobile, and the Web
 
 ambassa...@fourthworld.comhttp://www.FourthWorld.com

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Alex Tweedly via use-livecode]

I think regex (or multiple string comparisons) is too limited a way to go.

I would want to be able to do numeric comparisons, range comparisons, etc.

e.g. x is a number, x<25, 24 < x < 33, etc.

-- Alex.

On 26/08/2017 05:05, Brian Milby via use-livecode wrote:

tElement isn't the part that would change (you are still searching against
a single value in the array), you would need to modify the comparison to
allow for pNeedle to contain multiple values and the operators.  It would
probably be easier to modify to use a RegEx.  If you have 2 comma separated
items, you could use something like the following (I'm sure the more
experienced can offer much better ways though):

*if* (tElement contains item 1 of pNeedle) or (tElement contains item 2 of
pNeedle) *then*

On Fri, Aug 25, 2017 at 10:41 PM, Sannyasin Brahmanathaswami via
use-livecode  wrote:


OK I will bite

what would an array representing an "AND/OR"
style query look like such that if used as a predicate

for

  else if tElement contains pNeedle then

it would return true/false

How would you have to construct "tElement"

??



Mark: In terms of generalizing that function - then the key line which
checks
whether an element matches is:

  else if tElement contains pNeedle then


 One could imagine that this could be an arbitrary predicate - pNeedle
 doesn't have to be a string, so could be an array representing a AND /
 OR style query; or could be a regex (in which case you'd use
matchText).

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your
subscription preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode



___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Monte Goulding via use-livecode]

As a slightly related aside in this conversation I’d still one day like to add 
`each` to filter:

— filter array and retain elements where element[“foo”][“bar”] = “baz"
put “foo” into tPath[1]
put “bar” into tPath[2]
filter elements of tArray by evaluating each[tPath] is “baz”

Cheers

Monte
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Brian Milby via use-livecode]

tElement isn't the part that would change (you are still searching against
a single value in the array), you would need to modify the comparison to
allow for pNeedle to contain multiple values and the operators.  It would
probably be easier to modify to use a RegEx.  If you have 2 comma separated
items, you could use something like the following (I'm sure the more
experienced can offer much better ways though):

*if* (tElement contains item 1 of pNeedle) or (tElement contains item 2 of
pNeedle) *then*

On Fri, Aug 25, 2017 at 10:41 PM, Sannyasin Brahmanathaswami via
use-livecode  wrote:

> OK I will bite
>
> what would an array representing an "AND/OR"
> style query look like such that if used as a predicate
>
> for
>
>  else if tElement contains pNeedle then
>
> it would return true/false
>
> How would you have to construct "tElement"
>
> ??
>
>
>
> Mark: In terms of generalizing that function - then the key line which
> checks
> whether an element matches is:
>
>  else if tElement contains pNeedle then
>
>
> One could imagine that this could be an arbitrary predicate - pNeedle
> doesn't have to be a string, so could be an array representing a AND /
> OR style query; or could be a regex (in which case you'd use
> matchText).
>
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Brian Milby via use-livecode]

I created a stack to do some testing to see about making a function that
could do multiple types of tests.  What I found was that adding another if
at that point added about 6% to the search time and a second level (else
if) added another 10% (16% total).  (I added a parameter for search type -
"String".  I used "String2" to get a second option, but used the same
test).  When I used number instead of strings it was a little lower, but
not much (5.5% and 11% total)


> In terms of generalizing that function - then the key line which checks
> whether an element matches is:
>
> else if tElement contains pNeedle then
>
> One could imagine that this could be an arbitrary predicate - pNeedle
> doesn't have to be a string, so could be an array representing a AND / OR
> style query; or could be a regex (in which case you'd use matchText).
>
> Warmest Regards,
>
> Mark.
>
> --
> Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/
> LiveCode: Everyone can create apps
>
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Sannyasin Brahmanathaswami via use-livecode]

OK I will bite 

what would an array representing an "AND/OR" 
style query look like such that if used as a predicate 

for

 else if tElement contains pNeedle then

it would return true/false

How would you have to construct "tElement"

??



Mark: In terms of generalizing that function - then the key line which checks
whether an element matches is:

 else if tElement contains pNeedle then


One could imagine that this could be an arbitrary predicate - pNeedle 
doesn't have to be a string, so could be an array representing a AND / 
OR style query; or could be a regex (in which case you'd use matchText).

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Mark Waddingham via use-livecode]

On 2017-08-25 17:43, Sannyasin Brahmanathaswami via use-livecode wrote:

@ Mark Waddingham

Awesome! thanks, I will test this later…

Yes, my solution, at the urging of our Cohorts of the Round Table
LiveCode Data Diggers,
who earlier said "Don't go looking for your needles in arrays." (

I just reformulated the SQL quaryand passed it back "up" to the stack
script which fetched the original complete data set.
That worked, but there are many use cases where this will be a better 
solution.


in the context of songs  I guess we iterate several times through this
example to a case where


[title %like% "laughter" OR subtitle %like% "laughter" OR description
%like% "laughter"

AND

genre = "Songs By Nerds"

AND

theme = "Never Say Can't"


requires three trips to the mall:

array contains "Laughter"
  AND
array contains " Songs By Nerds "
  AND
array contains "Never Say Can't"

But these are very small data sets, relatively speaking.. so it will
be fast.  But what serialization results is unclear…


Which is better (using arrays, or the DB direct) largely depends on the 
size of the dataset.


If you have a very large dataset, then you are going to run out of 
memory if you try and load it all into RAM - so a better approach (as 
you have done) is to bind the search to SQL queries on the DB. This is 
very scalable; and if you back everything you do with the (large) 
dataset by the DB, you get 'only needing what you need in RAM at any one 
time' essentially for 'free' (although, at the expense of needing to 
abstract through SQL I suppose).


If you have a small dataset (which isn't going to grow, or only grow 
very slowly) then it probably makes no difference, and you are better 
off using whichever is easier to code.


In terms of generalizing that function - then the key line which checks 
whether an element matches is:


else if tElement contains pNeedle then

One could imagine that this could be an arbitrary predicate - pNeedle 
doesn't have to be a string, so could be an array representing a AND / 
OR style query; or could be a regex (in which case you'd use matchText).


Warmest Regards,

Mark.

--
Mark Waddingham ~ m...@livecode.com ~ http://www.livecode.com/
LiveCode: Everyone can create apps

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Sannyasin Brahmanathaswami via use-livecode]

@ Mark Waddingham

Awesome! thanks, I will test this later… 

Yes, my solution, at the urging of our Cohorts of the Round Table LiveCode Data 
Diggers, 
who earlier said "Don't go looking for your needles in arrays." (

I just reformulated the SQL quaryand passed it back "up" to the stack script 
which fetched the original complete data set.
That worked, but there are many use cases where this will be a better solution.

in the context of songs  I guess we iterate several times through this example 
to a case where


[title %like% "laughter" OR subtitle %like% "laughter" OR description %like% 
"laughter" 

AND

genre = "Songs By Nerds"

AND

theme = "Never Say Can't"


requires three trips to the mall:

array contains "Laughter"  
  AND
array contains " Songs By Nerds "  
  AND
array contains "Never Say Can't"  

But these are very small data sets, relatively speaking.. so it will be fast.  
But what serialization results is unclear… 




 

On 8/24/17, 11:03 PM, "use-livecode on behalf of Mark Waddingham via 
use-livecode"  wrote:

I think you may have already found a solution to what you want to do... 
However, you'll find my version below.

The arrayFindElementsContainingString command is a generic handler which 
will search all elements of an array recursively for containing pNeedle.

It returns a sequence (numerically indexed array) of array paths 
containing the needle.

The handler exploits the 'dynamic path' lookup feature of arrays. If you 
construct a sequence (numerically keyed array starting at 1) array of 
strings, you can use that sequence in [ ... ], and the engine will treat 
it as a sequence of keys. e.g.

   put "foo" into tPath[1]
   put "baz" into tPath[2]
   put tArrayA[tPath] into field "Result"

Here the last line is equivalent to tArrayA["foo"]["baz"].

(Note this all works on arrays and strings, the test handler uses Json 
as a convenient way to represent an array in a field!)

 TEST HANDLER
-- Requires a field "Results"
-- Requires a field "Json" containing the JSON form of the array to be 
searched
-- Requires a field "Needle" containing the string to search for
-- Requires a field "Results"

/* Test the array finding */
command testArrayFind pNeedle
/* Clear the results field */
put empty into field "Results"

/* Get an array which is JSON encoded in field "Json" as an array */
local tArray
put JSONImport(field "Json") into tArray

/* Search the array for the needle, fetched from field "Needle" */
arrayFindElementsContainingString tArray, field "Needle"

/* Iterate over each found path */
repeat for each element tFoundPath in it
   /* Each path is a numerically keyed array (sequence), these can be 
used
   * directly as array indicies, the engine will iterate through the 
elements of
   * the sequence to get the keys to use */
   local tValue
   put tArray[tFoundPath] into tValue

   /* Create a slash delimited list of keys (note - this won't work 
if any of the
   * keys in the path contain '/'!). */
   combine tFoundPath with "/"

   /* Show the list of results in field "Results" in the form:
   * slash delimited path : value
   */
   put tFoundPath & ":" & tValue & return after field "Results"
end repeat
end testArrayFind

 LIBRARY FUNCTIONALITY

/* Search all elements of pArray for containment of pNeedle. The
  * command returns a numerically keyed array in 'it', each element of
  * which is an array path to an element containing pNeedle. */
command arrayFindElementsContainingString pArray, pNeedle
/* We expect an array for pArray */
if pArray is not an array and pArray is not empty then
   throw "pArray must be an array"
end if

/* We expect a string for pNeedle */
if pNeedle is an array then
   throw "pNeedle must be a string"
end if

/* As arrays are recursive, we pass through the current base path
 * for the sub-array being searched to an auxillary private command 
*/
local tBasePath
put empty into tBasePath

/* The auxillary private command accumulates complete paths in
 * tPaths, which is a numerically keyed array (sequence). */
local tPaths
put empty into tPaths
_arrayFindElementsContainingString pArray, pNeedle, tBasePath, tPaths

/* Using 'return for value' returns the value in 'it' in the caller. 
*/
return tPaths for value
end arrayFindElementsContainingString

private command _arrayFindElementsContainingString pArray, 

Re: Search Values of Array in "One Go"

[Mark Waddingham via use-livecode]

On 2017-08-23 17:24, Sannyasin Brahmanathaswami via use-livecode wrote:

We use these arrays with media metadata that is extracted from our
database of media metadata. The dbase was originally designed for lots
of columns so that "there is nothing we cannot know about a media
item" incorporating DCIM columns and also W3C media metadata
initiative's recommended properties.  In actually usage for any give
media item only a small subset of columns containe any data. but we do
use almost all the columns at one time or another.  There are no
blobs, so this is very light weight data.

Anyway… this results in arrays which contain 1 top level key per
record and that element contains another 40 or so keys for the columns
of the database. most of which are empty/unused. So these arrays are
never "heavy" in terms of bytes.


I think you may have already found a solution to what you want to do... 
However, you'll find my version below.


The arrayFindElementsContainingString command is a generic handler which 
will search all elements of an array recursively for containing pNeedle.


It returns a sequence (numerically indexed array) of array paths 
containing the needle.


The handler exploits the 'dynamic path' lookup feature of arrays. If you 
construct a sequence (numerically keyed array starting at 1) array of 
strings, you can use that sequence in [ ... ], and the engine will treat 
it as a sequence of keys. e.g.


  put "foo" into tPath[1]
  put "baz" into tPath[2]
  put tArrayA[tPath] into field "Result"

Here the last line is equivalent to tArrayA["foo"]["baz"].

(Note this all works on arrays and strings, the test handler uses Json 
as a convenient way to represent an array in a field!)


 TEST HANDLER
-- Requires a field "Results"
-- Requires a field "Json" containing the JSON form of the array to be 
searched

-- Requires a field "Needle" containing the string to search for
-- Requires a field "Results"

/* Test the array finding */
command testArrayFind pNeedle
   /* Clear the results field */
   put empty into field "Results"

   /* Get an array which is JSON encoded in field "Json" as an array */
   local tArray
   put JSONImport(field "Json") into tArray

   /* Search the array for the needle, fetched from field "Needle" */
   arrayFindElementsContainingString tArray, field "Needle"

   /* Iterate over each found path */
   repeat for each element tFoundPath in it
  /* Each path is a numerically keyed array (sequence), these can be 
used
  * directly as array indicies, the engine will iterate through the 
elements of

  * the sequence to get the keys to use */
  local tValue
  put tArray[tFoundPath] into tValue

  /* Create a slash delimited list of keys (note - this won't work 
if any of the

  * keys in the path contain '/'!). */
  combine tFoundPath with "/"

  /* Show the list of results in field "Results" in the form:
  * slash delimited path : value
  */
  put tFoundPath & ":" & tValue & return after field "Results"
   end repeat
end testArrayFind

 LIBRARY FUNCTIONALITY

/* Search all elements of pArray for containment of pNeedle. The
 * command returns a numerically keyed array in 'it', each element of
 * which is an array path to an element containing pNeedle. */
command arrayFindElementsContainingString pArray, pNeedle
   /* We expect an array for pArray */
   if pArray is not an array and pArray is not empty then
  throw "pArray must be an array"
   end if

   /* We expect a string for pNeedle */
   if pNeedle is an array then
  throw "pNeedle must be a string"
   end if

   /* As arrays are recursive, we pass through the current base path
* for the sub-array being searched to an auxillary private command 
*/

   local tBasePath
   put empty into tBasePath

   /* The auxillary private command accumulates complete paths in
* tPaths, which is a numerically keyed array (sequence). */
   local tPaths
   put empty into tPaths
   _arrayFindElementsContainingString pArray, pNeedle, tBasePath, tPaths

   /* Using 'return for value' returns the value in 'it' in the caller. 
*/

   return tPaths for value
end arrayFindElementsContainingString

private command _arrayFindElementsContainingString pArray, pNeedle, 
pBasePath, @xFoundPaths

   repeat for each key tKey in pArray
  /* Fetch the value of the key */
  local tElement
  put pArray[tKey] into tElement

  /* Create a new path from the base path by appending the current 
key */

  local tPath
  put pBasePath into tPath
  put tKey into tPath[the number of elements in tPath + 1]

  /* What we do depends on the content of the element */
  if tElement is an array then
 /* If the element is an array, then we recurse passing through 
the path to this key

  * as the base path */
 _arrayFindElementsContainingString tElement, pNeedle, tPath, 
xFoundPaths

  else if tElement contains pNeedle then
 /* If the 

Re: Search Values of Array in "One Go"

[Bob Sneidar via use-livecode]

This will work with any array. It recursively calls itself as it encounters 
arrays within arrays. I agree with Richard that if all you want is one shot at 
an array to find the element(s) you are looking for, then this may (or may not) 
be a cleaner approach, but I think if you are taking multiple passes at 
finding/filtering with an array, this will at least get you there. If filtering 
with multiple passes, remember to put the text of the array in another variable 
 as filter *replaces* the contents of the source variable with the results. 

Here is the function. Thinking about it, I suppose you could modify this to 
include a filter parameter so that only the keys matching the filter would be 
returned. 

Bob S


function altPrintKeys @pArray, theKeyList
put numtochar(11) into vertTab
put numtochar(30) into altCr
put the keys of pArray into theKeys
sort theKeys numeric

repeat FOR each line theKey in theKeys
put "[" & theKey & "] " after theKeyList
if theKey is not a number then
replace "[" & theKey & "]" WITH "[" & quote & theKey & quote & "]" 
in theKeyList
end if
if pArray[theKey] is an array then
put pArray[theKey] into theTempArray
put altPrintKeys(theTempArray, theKeyList, pFullData) after theText
put empty into the last word of theKeyList
delete the last char of theKeyList
put cr into the last char of theText
else
put "pArray " & the last word of theKeyList into theKeyName
-- put "put " & theKeyName & " into theValue" into theCommand
-- do theCommand
put value(theKeyName) into theValue
replace tab WITH vertTab in theValue
replace return WITH altCr in theValue
put theKeyList & tab & theValue & comma after theText
put empty into the last word of theKeyList
delete the last char of theKeyList
end if
end repeat

return theText
end altPrintKeys

> On Aug 24, 2017, at 12:14 , Sannyasin Brahmanathaswami via use-livecode 
>  wrote:
> 
> what about a 2 diminsional array


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Richard Gaskin via use-livecode]

Bob Sneidar wrote:

> I suppose thinking about it you could arrayencode an array then do a
> search on it

You probably don't want to do that.  I've given a lot of thought to 
methods of indexing large LSON files on disk, and I can find no method 
anywhere near as practical as working with them in their native array 
structure, given their linear, non-indexed LSON format.


Besides, in order to serialize an LC array it needs to be small enough 
to fit into RAM to begin with, so the one thing we know about all LSON 
files is that they fit into RAM nicely. :)


It's been a while since Mark Waddingham generously provided some notes 
in the LSON format (for the older format, which has changed since the 
introduction of Unicode in v7), but IIRC it was roughly:


0x05 -- one-bye header indicating that what follows is an array (now 
0x06 in v7 and later)

  
  NULL <4-byte data length indicator>

There is a different op-code for numbers than for strings (IIRC 0x02 for 
numbers), allowing numbers to use a more compact binary form.


I'm guessing that in addition to the new op-code indicating an array 
type, the former NULL separator between element name and length UINT4 
has been replaced with a preceding length byte, since of course NULLs 
can be part of the Unicode string of the element name.


Nice tidy format, well suited for disk storage and network transfer, but 
looking for things requires linear search in LSON, whereas the 
de-serialized native array form takes advantage of the super-quick 
bucket hash to find a given key.


Much as XQuery works on an in-memory, already-parsed form of XML, 
searching associative arrays in memory will be the way to go.


So looping is both faster to execute and easier to script in array form.

The biggest challenges would be parsing the query expression, and 
generalizing evaluation within the loop(s) to handle the range of query 
options.



--
 Richard Gaskin
 Fourth World Systems
 Software Design and Development for the Desktop, Mobile, and the Web
 
 ambassa...@fourthworld.comhttp://www.FourthWorld.com

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Sannyasin Brahmanathaswami via use-livecode]

Bob: fascinating

[1] ["addr2"]   ,[1] ["smtppassword"]   none,[1] ["itemail"],[1] 
["contactphone"]   661-257-1177,[1] ["contactonsite"]  false, etc. 

what about a 2 diminsional array



You can see from this format that you can convert to lines and then you can 
filter the lines by key or by value. If you want by key you could filter by 
quote & key & quote. Otherwise for values filter by tab & value. 

The downside would be working with really large arrays because of course 
this loops through every array element to produce a textual representation of 
the array. Still, it's something if you already have an array for one reason or 
another, and you want to search/filter the array for keys or values.

BR: well I am now going back to the dbase, because it was so easy just to pass 
a new query array (name value pairs for the columns to look up) but on android 
the dBase lib and look up is slow on Android.  I wonder if doing something like 
youa re doing with the array that is already in memory would be faster.

I will have to figure out exactly what you are doing first… can you post the 
full functions?

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Bob Sneidar via use-livecode]

I've partially deciphered it. I created a tab delimited table of each 
non-printing character followed by the text between that and the next 
non-printing character. Here is a sample:

12  contactphone
8   
0   
12  661-257-1177
0   
13  contactonsite
8   
0   
5   false
0   
12  contactemail
7   
0   
6   itname
7   
0   

The 12 before the contactphone and the 5 before the false are clearly the 
length of the value. 0 is some kind of delimiter. 8 following contactphone and 
contactonsite appears to mean "what follows is a value" (also preceded by a 
length) and 7 means "empty". 

What is clear before going any further is that a key cannot be discerned from 
this format. At least not very easily. 

Now I wrote a function years ago similar to printKeys() but specifically 
designed so that filter could be used on the result. It produces a comma 
delimited list, each item of which is a tab separated value of the array key 
and the value. It looks like this:

[1] ["addr2"]   ,[1] ["smtppassword"]   none,[1] ["itemail"],[1] 
["contactphone"]   661-257-1177,[1] ["contactonsite"]  false, etc. 

You can see from this format that you can convert to lines and then you can 
filter the lines by key or by value. If you want by key you could filter by 
quote & key & quote. Otherwise for values filter by tab & value. 

The downside would be working with really large arrays because of course this 
loops through every array element to produce a textual representation of the 
array. Still, it's something if you already have an array for one reason or 
another, and you want to search/filter the array for keys or values. 

Bob S



___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Bob Sneidar via use-livecode]

Okay it's not entirely straight forward. I took a simple array from a datagrid 
with one record and ran this on it:

on mouseUp pButtonNumber
   put the dgData of group "dgsites" into aSiteData
   put arrayencode(aSiteData) into tSiteData
   repeat for each char tChar in tSiteData
  add 1 to tCounter
  if cleanAscii(tChar) is not empty then next repeat
  put tCounter & tab & charToNum(tChar) & cr after tAscii
   end repeat
   set the itemDelimiter to tab
   sort lines of tAscii numeric by item 2 of each
   put tAscii
end mouseUp

This returns a tab delimited list of position and non-printing ascii characters 
ranging from 0 to 58. What the ascii characters designate is the mystery. Any 
documentation on that? 

Bob S


> On Aug 24, 2017, at 10:50 , Bob Sneidar via use-livecode 
>  wrote:
> 
> I suppose thinking about it you could arrayencode an array then do a search 
> on it, but I am not sure how you would discern the array keys. I'll have to 
> look at the encoded format. At that point if that approach worked, the only 
> advantage to searching a memory based array is you would be skipping the 
> encoding step, and searching the actual array in memory with the engine might 
> be considerably faster than having LC search a text variable. 
> 
> I'll play around with it to see what I can come up with. 
> 
> Bob S


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Bob Sneidar via use-livecode]

I mentioned a few years ago when arrays first were introduced that there ought 
to be an engine based way to seach arrays for values, returning matching key 
references (not just one but ALL). If you are only interested in certain keys 
you could then filter the results. 

To make this fast for large arrays, you might have to have a way to index the 
arrays, but now we are talking about a memory based database system, and I am 
not sure Livecode arrays are the way to go. It may be enough to be able to do a 
kind of full text search on an array in memory. 

I suppose thinking about it you could arrayencode an array then do a search on 
it, but I am not sure how you would discern the array keys. I'll have to look 
at the encoded format. At that point if that approach worked, the only 
advantage to searching a memory based array is you would be skipping the 
encoding step, and searching the actual array in memory with the engine might 
be considerably faster than having LC search a text variable. 

I'll play around with it to see what I can come up with. 

Bob S


> On Aug 24, 2017, at 09:00 , Richard Gaskin via use-livecode 
>  wrote:
> 
> At my local Linux User Group last year a fella from NASA gave a presentation 
> about a system he'd developed for querying planetary data via URLs.
> 
> The data is huge (basically everything every interplanetary satellite/robot 
> delivers to JPL) and his choice of HDF (Hierarchical Data Format) as the 
> storage base was interesting in itself (imagine a large encoded array but 
> designed for traversal on disk).
> 
> But even more so was his query API.  A thing of beauty.  One-line URLs could 
> be entered in a browser to locate just about anything in the massive HDF 
> store.
> 
> Ultimately it could be seen as a simplified form of XQuery, every bit as 
> elegant and flexible but without the XML-specific cruft (having to account 
> for the mix of elements and attributes).
> 
> Of course this led me to thinking of ways to use this for LC arrays.
> 
> It should be possible to craft a library to allow XQuery-like one-liners to 
> locate anything anywhere in any array.
> 
> It should also be possible to do this in the engine with blindingly-faster 
> efficiency.
> 
> But as a starting point, fleshing out the API in script seemed at least a 
> good first step.
> 
> I do not have the time to pursue it in earnest.  But XQuery is well 
> documented, and a sufficiently-inspired person may be able to come up with a 
> pretty great query system given sufficient time to explore the 
> possibilities...
> 
> -- 
> Richard Gaskin
> Fourth World Systems
> Software Design and Development for the Desktop, Mobile, and the Web


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Richard Gaskin via use-livecode]

Sannyasin Brahmanathaswami wrote:

> OK so let's say user  want sto search "blues"  and I want the UX to
> check *all* columns in the array.  What is the best way to do this?
...
> Is there a more efficient approach?

At my local Linux User Group last year a fella from NASA gave a 
presentation about a system he'd developed for querying planetary data 
via URLs.


The data is huge (basically everything every interplanetary 
satellite/robot delivers to JPL) and his choice of HDF (Hierarchical 
Data Format) as the storage base was interesting in itself (imagine a 
large encoded array but designed for traversal on disk).


But even more so was his query API.  A thing of beauty.  One-line URLs 
could be entered in a browser to locate just about anything in the 
massive HDF store.


Ultimately it could be seen as a simplified form of XQuery, every bit as 
elegant and flexible but without the XML-specific cruft (having to 
account for the mix of elements and attributes).


Of course this led me to thinking of ways to use this for LC arrays.

It should be possible to craft a library to allow XQuery-like one-liners 
to locate anything anywhere in any array.


It should also be possible to do this in the engine with 
blindingly-faster efficiency.


But as a starting point, fleshing out the API in script seemed at least 
a good first step.


I do not have the time to pursue it in earnest.  But XQuery is well 
documented, and a sufficiently-inspired person may be able to come up 
with a pretty great query system given sufficient time to explore the 
possibilities...


--
 Richard Gaskin
 Fourth World Systems
 Software Design and Development for the Desktop, Mobile, and the Web
 
 ambassa...@fourthworld.comhttp://www.FourthWorld.com

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Richard Gaskin via use-livecode]

Sannyasin Brahmanathaswami wrote:
> I can't wait to use the new tools for watching processes to see where
> > bottle necks arise…
...
> did you see the pitch for business upgrade?  scroll down.. there some
> new thing that give you time on processes

Thanks.  I'd forgotten that was a new premium feature.

MetaCard included a profiler.

Might be nice if someone had time to update it for use in LC Community.

--
 Richard Gaskin
 Fourth World Systems
 Software Design and Development for the Desktop, Mobile, and the Web
 
 ambassa...@fourthworld.comhttp://www.FourthWorld.com

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Sannyasin Brahmanathaswami via use-livecode]

Well thanks to all of you urging me to stay away from re-processing the array 
but going back to the dbase.

I Looked back at my code and realized that I *had* set it up to be re-usable, I 
just forgot how well I had done that 

I have this function in the behavior_Listen  which is attached to the stack

local sPlayListQueryA

command playlist_FetchData

# this handler gets everything we need from the dbase and assumes the column 
values for the query
# are set in the sPlayListQueryA

-

duh!  I had written everything I needed already!

I only needws to add this

on playList_FetchData pQueryArray
   
   if pQueryArray is not empty then # we are passing a query from the search 
behavior
  put pQueryArray into sPlaylistQueryA
   end if
   
and away we go, exact code to instantiate the entire play list of all audio 
items for that class is now reused to just fetch what the user wants.
Awesome… this  all I needed in my search behavior  was to set up  the query 
(values for column look ups)

 put tGenre into tFilterListQueryA["genre"]
 put tTheme into tFilterListQueryA["theme"]
 put tStringToFind into tFilterListQueryA["title"]
put tStringToFind into tFilterListQueryA["subtitle"]
put tStringToFind into tFilterListQueryA["description"]
put tStringToFind into tFilterListQueryA["artists"]

   revealMoreActions "false"  # hide the form and fetch a new list from the 
dbase
   playList_fetchData tFilterListQueryA

all the discussion on speed seems mute in this context as the data sets are 
small an it happens in "the flash of an eye"

Thanks gentlemen, you saved me from going off the deep end re-parsing the local 
array!



BR



On 8/23/17, 2:47 PM, "use-livecode on behalf of Mike Bonner via use-livecode" 
 wrote:

I think you need to use ":memory:" as the name actually.  It looks (from
the sqlite docs) that supplying no name creates a temporary db file.
And now i'm curious.. Is there a way to attach the file based db to a
memory db?

IE Open the file database, then use the info here:
https://www.sqlite.org/lang_attach.html and attach a memory database.

Then (way over my head, but I seem to recall reading somewhere..) that one
can use the results of a select to create a temporary table..  If
interested, you might look here under create table as select...
http://sqlite.org/lang_createtable.html

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Mike Bonner via use-livecode]

I think you need to use ":memory:" as the name actually.  It looks (from
the sqlite docs) that supplying no name creates a temporary db file.
And now i'm curious.. Is there a way to attach the file based db to a
memory db?

IE Open the file database, then use the info here:
https://www.sqlite.org/lang_attach.html and attach a memory database.

Then (way over my head, but I seem to recall reading somewhere..) that one
can use the results of a select to create a temporary table..  If
interested, you might look here under create table as select...
http://sqlite.org/lang_createtable.html

On Wed, Aug 23, 2017 at 6:31 PM, Sannyasin Brahmanathaswami via
use-livecode  wrote:

> Yep… I just found that too.
>
> and related 4-yr old discussion
>
> http://use-livecode.runrev.narkive.com/ntBLeXGM/relative-
> performance-of-two-dimensional-array-and-in-memory-sqlite-database
>
>
> What I don't get is
>
>  "don't give it a name"open/create in memory,
>
> but the data is on SQLite dbase on disk, so you *have* to read it one way
> or another.
>
> (scratch his head…)
>
>
> On 8/23/17, 2:13 PM, "use-livecode on behalf of Alex Tweedly via
> use-livecode"  use-livecode@lists.runrev.com> wrote:
>
> On 24/08/2017 01:04, Sannyasin Brahmanathaswami via use-livecode wrote:
> >2) "create an in-memory SQLite Database."
> >
> >   this is news! Where do we find docs on this? right now the SQLite
> data base is in
> >
> >   specialFolderPath("documents")/jnanam.sqlite
> >
> >how do you "create in memory?
> >
> >
> According to an old message (i.e. I've not tried it :-):
>
> On Fri, Jun 15, 2012 at 5:21 PM, Bob Sneidar  http://lists.runrev.com/mailman/listinfo/use-livecode>> wrote:
>
> //>//>/On Jun 15, 2012, at 4:13 PM, Mike Bonner wrote: />//>/ > Don't
> specify a database name and it will open the db in memory. />/ > />/ > get
> revopendatabase("sqlite",,) --not sure how many commas off the />/top
> />/ > of my head. />/ > />/ > On Fri, Jun 15, 2012 at 5:08 PM, Bob Sneidar
>  > wrote: />/ >
> />/ >> I am just certain that I read on this forum that you could create an
> />/ >> sqLite database in memory! But when looking through the dictionary
> and />/in />/ >> the LiveCode lessons there seems to be no mention of how!
> Can someone />/point />/ >> me to the right docs for this? />/ >> />/ >>
> Bob />/ >> />/ >> /
>
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Sannyasin Brahmanathaswami via use-livecode]

Yep… I just found that too.

and related 4-yr old discussion

http://use-livecode.runrev.narkive.com/ntBLeXGM/relative-performance-of-two-dimensional-array-and-in-memory-sqlite-database


What I don't get is

 "don't give it a name"open/create in memory, 

but the data is on SQLite dbase on disk, so you *have* to read it one way or 
another.

(scratch his head…) 
 

On 8/23/17, 2:13 PM, "use-livecode on behalf of Alex Tweedly via use-livecode" 
 wrote:

On 24/08/2017 01:04, Sannyasin Brahmanathaswami via use-livecode wrote:
>2) "create an in-memory SQLite Database."
>
>   this is news! Where do we find docs on this? right now the SQLite data 
base is in
>
>   specialFolderPath("documents")/jnanam.sqlite
>
>how do you "create in memory?
>
>
According to an old message (i.e. I've not tried it :-):

On Fri, Jun 15, 2012 at 5:21 PM, Bob Sneidar http://lists.runrev.com/mailman/listinfo/use-livecode>> wrote:

//>//>/On Jun 15, 2012, at 4:13 PM, Mike Bonner wrote: />//>/ > Don't 
specify a database name and it will open the db in memory. />/ > />/ > get 
revopendatabase("sqlite",,) --not sure how many commas off the />/top />/ > 
of my head. />/ > />/ > On Fri, Jun 15, 2012 at 5:08 PM, Bob Sneidar http://lists.runrev.com/mailman/listinfo/use-livecode>> wrote: />/ > />/ 
>> I am just certain that I read on this forum that you could create an />/ >> 
sqLite database in memory! But when looking through the dictionary and />/in 
/>/ >> the LiveCode lessons there seems to be no mention of how! Can someone 
/>/point />/ >> me to the right docs for this? />/ >> />/ >> Bob />/ >> />/ >> /

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Alex Tweedly via use-livecode]

On 24/08/2017 01:04, Sannyasin Brahmanathaswami via use-livecode wrote:

2) "create an in-memory SQLite Database."

  this is news! Where do we find docs on this? right now the SQLite data base 
is in

  specialFolderPath("documents")/jnanam.sqlite

how do you "create in memory?



According to an old message (i.e. I've not tried it :-):

On Fri, Jun 15, 2012 at 5:21 PM, Bob Sneidar http://lists.runrev.com/mailman/listinfo/use-livecode>> wrote:

//>//>/On Jun 15, 2012, at 4:13 PM, Mike Bonner wrote: />//>/ > Don't specify a database name and it will open the db in memory. />/ > />/ > get revopendatabase("sqlite",,) --not sure how many commas off the />/top />/ > of my head. />/ > />/ > On Fri, Jun 15, 2012 at 5:08 PM, Bob Sneidar > wrote: />/ > />/ >> I am just certain that I read on this forum that you could create an />/ >> sqLite database in memory! But when looking through the dictionary and />/in />/ >> the LiveCode lessons there seems to be no mention of how! Can someone />/point />/ >> me to the right docs for this? />/ >> />/ >> Bob />/ >> />/ >> /



-- Alex.
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Sannyasin Brahmanathaswami via use-livecode]

Alex, thanks, this is very helpful

1) put x into aFilteredPlayListIndex[z]
(there's no need to replicate all the data when the array will continue to 
exist, just keep the indices you want).

awesome… I think I could increment the  z from 1  up and then my clickline 
number will still work.

2) "create an in-memory SQLite Database."

 this is news! Where do we find docs on this? right now the SQLite data base is 
in

 specialFolderPath("documents")/jnanam.sqlite

how do you "create in memory?


 

On 8/23/17, 9:06 AM, "Alex Tweedly via use-livecode" 
 wrote:

I'm not going to disagree with the other replies 

Plan A.   use your database

Plan B.   create an in-memory SQLite database.

Plan C. use an array, kind of like you are doing.

And *only* if there are good reasons for not doing Plans A or B, then :

C1.  Instead of

repeat for each key x in aPlaylist
repeat for each key y in aPlayList[x]


do

repeat for each key x in aPlaylist
repeat for each element ey in aPlayList[x]

Also, instead of
 put aPlayList[x] into aFilteredPlayList[z]

do
 put x into aFilteredPlayListIndex[z]
(there's no need to replicate all the data when the array will continue to 
exist, just keep the indices you want).


C2. Did you really mean "check *all* columns in the array" ?
Should it not be "check all the 'properties' columns" ?
If so - that's another reason to use a/the database :-)

C3. Create an 'allproperties' column ?
There's probably a fast/easy way to do that either in the database, or as 
you extract from it - but a database expert would have to tell us how to do 
that.

Alternatively, when you extract, just do

repeat for each key x in aPlayList
   repeat for each item y in thePropertyColumns
   put aPlayList[x][y] & SPACE after aPlayList[x]["allproperties"]
   end repeat
end repeat

or, delay this until first search :

repeat for each key x in aPlayList
if aPlayList[x]["allproperties"] is empty then
   repeat for each item y in thePropertyColumns
   put aPlayList[x][y] & SPACE after aPlayList[x]["allproperties"]
   end repeat
   if aPlayList[x]["allproperties"] is empty then -- so that subsequent 
searches never rebuild this
 put SPACE into aPlayList[x]["allproperties"]
   end if
end if
if aPlayList[x]["allproperties"] contains pFilterPattern then
  put x & CR after z
end if
end repeat

  
Alex.

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Sannyasin Brahmanathaswami via use-livecode]

The media meta dbase "jnanam.item" table actually has  87 columns, ha! way over 
kill, but I'm actually glad we did that because there is indeed "nothing we 
cannot know about any media item in the universe"

But  any given item… only uses a subset. 



media_type = "audio/song"

and

media_type = "audio/inspiredTalk"

are the context where users need search. 

will only use a subset.. but I _do_have all 87 props in the array.

For this use case we are only looking at/"into"

item.media_type
item.title
item.sub_title
item.approved_for_public 
item.file_id (a varchar text string)
item.description
item.artists
item.date_entered # added to the dbase - allow us to fetch "recent additions"
item.genre
item.theme
item.transcription

So radio buttons will give the user the genre and theme "enum" values  to 
choose from, but the search string can't be constrained..we need to check into

item.title
item.sub_title
item.description
item.transcription.

do the look up into the array will first filter by genre and theme and *then 
check inside the above three… if "all" is check in genre and theme then we look 
for the string in all three var char fields title, sub_title, description.

Hmm audio/songs is only running at 400 records (= 
sSelectionsA[audiofiles][1-400]  

that will go up after I open the gate for user generated uploads

audio/inspired_talks  is where we start hitting 1000+ records… and it "gets 
worse" there because we need to start looking into

item.transcription  = long text for 30 minute talk…










On 8/23/17, 10:04 AM, "use-livecode on behalf of Alex Tweedly via use-livecode" 
 wrote:


3 columns ?  I thought it was :
>and 30+  more "properties" for the media item
So, 3 or 30+ ? Probably make a difference ...

Alex.



___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Sannyasin Brahmanathaswami via use-livecode]

did you see the pitch for business upgrade?  scroll down.. there some new thing 
that give you time on processes

 

On 8/23/17, 9:28 AM, "use-livecode on behalf of Richard Gaskin via 
use-livecode"  wrote:

> I can't wait to use the new tools for watching processes to see where
> bottle necks arise…

?



___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Alex Tweedly via use-livecode]

On 23/08/2017 20:07, Sannyasin Brahmanathaswami via use-livecode wrote:

Bob, Mike,

thanks…"use the dbase" good thought
...

So then it comes down to whether doing a query semi-complex query that looks at 
50,000 records and uses  %like% against 3 columns is faster than iterating 
loops on a 1000 key array already in memory.


3 columns ?  I thought it was :

and 30+  more "properties" for the media item

So, 3 or 30+ ? Probably make a difference ...

Alex.

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Richard Gaskin via use-livecode]

Sannyasin Brahmanathaswami wrote:

> I can't wait to use the new tools for watching processes to see where
> bottle necks arise…

?

--
 Richard Gaskin
 Fourth World Systems
 Software Design and Development for the Desktop, Mobile, and the Web
 
 ambassa...@fourthworld.comhttp://www.FourthWorld.com

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Bob Sneidar via use-livecode]

Using equal queries is efficient *if* the columns are indexed. If you use LIKE, 
I do not believe that the indexes are used, unless you employ some kind of full 
text search indexing. Only certain databases support that. 

Bob S


> On Aug 23, 2017, at 12:07 , Sannyasin Brahmanathaswami via use-livecode 
>  wrote:
> 
> At least for theme and genre these are direct "hits" because there is an 
> "equals" look up against these two enum values; but the searchString look up 
> will still need to use   %like%  for  "blues" is in "title, subtitle and 
> description" . I have to check my sql query hand book aka "the internets" to 
> see if there is an efficient way to write that query.
> 
> So then it comes down to whether doing a query semi-complex query that looks 
> at 50,000 records and uses  %like% against 3 columns is faster than iterating 
> loops on a 1000 key array already in memory.
> 
> I can't wait to use the new tools for watching processes to see where bottle 
> necks arise…

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Sannyasin Brahmanathaswami via use-livecode]

Bob, Mike, 

thanks…"use the dbase" good thought

The reason we use the array is once we load that into a local than I use the 
click line number of the song list (titles in a simple list field)  to get the 
data for that particular song.  

click on line 9

get use everything we need from sSelectionsA ["audiofiles"]["9"]  
   -- title
   -- URL to the mp3 on the server to set for the player or for a download later
   -- artist

   etc..

But you make a good case ifwe are going back to get a new list, onemight as 
well just use the query params and just utllize that original handler that 
queries the dBase for a playlist in the first place.

At least for theme and genre these are direct "hits" because there is an 
"equals" look up against these two enum values; but the searchString look up 
will still need to use   %like%  for  "blues" is in "title, subtitle and 
description" . I have to check my sql query hand book aka "the internets" to 
see if there is an efficient way to write that query.

So then it comes down to whether doing a query semi-complex query that looks at 
50,000 records and uses  %like% against 3 columns is faster than iterating 
loops on a 1000 key array already in memory.

I can't wait to use the new tools for watching processes to see where bottle 
necks arise…



___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Alex Tweedly via use-livecode]

I'm not going to disagree with the other replies 

Plan A.   use your database

Plan B.   create an in-memory SQLite database.

Plan C. use an array, kind of like you are doing.

And *only* if there are good reasons for not doing Plans A or B, then :

C1.  Instead of

repeat for each key x in aPlaylist
   repeat for each key y in aPlayList[x]
   


do

repeat for each key x in aPlaylist
   repeat for each element ey in aPlayList[x]

Also, instead of
put aPlayList[x] into aFilteredPlayList[z]

do
put x into aFilteredPlayListIndex[z]
(there's no need to replicate all the data when the array will continue to 
exist, just keep the indices you want).


C2. Did you really mean "check *all* columns in the array" ?
Should it not be "check all the 'properties' columns" ?
If so - that's another reason to use a/the database :-)

C3. Create an 'allproperties' column ?
There's probably a fast/easy way to do that either in the database, or as you 
extract from it - but a database expert would have to tell us how to do that.

Alternatively, when you extract, just do

repeat for each key x in aPlayList
  repeat for each item y in thePropertyColumns
  put aPlayList[x][y] & SPACE after aPlayList[x]["allproperties"]
  end repeat
end repeat

or, delay this until first search :

repeat for each key x in aPlayList
   if aPlayList[x]["allproperties"] is empty then
  repeat for each item y in thePropertyColumns
  put aPlayList[x][y] & SPACE after aPlayList[x]["allproperties"]
  end repeat
  if aPlayList[x]["allproperties"] is empty then -- so that subsequent 
searches never rebuild this
put SPACE into aPlayList[x]["allproperties"]
  end if
   end if
   if aPlayList[x]["allproperties"] contains pFilterPattern then
 put x & CR after z
   end if
end repeat

 
Alex.



On 23/08/2017 16:24, Sannyasin Brahmanathaswami via use-livecode wrote:

We use these arrays with media metadata that is extracted from our database of media 
metadata. The dbase was originally designed for lots of columns so that "there is 
nothing we cannot know about a media item" incorporating DCIM columns and also W3C 
media metadata initiative's recommended properties.  In actually usage for any give media 
item only a small subset of columns containe any data. but we do use almost all the 
columns at one time or another.  There are no blobs, so this is very light weight data.

Anyway… this results in arrays which contain 1 top level key per record and that element 
contains another 40 or so keys for the columns of the database. most of which are 
empty/unused. So these arrays are never "heavy" in terms of bytes.

e.g.

aPlayList  may have 1000 keys (numeric 1-1000)

where each top level key contains a whole record:

in each one we have

aPlayList [1]["title"]
aPlayList [1]["subtitle"]
aPlayList [1]["genre"]
aPlayList [1]["description"]
aPlayList [1]["artist"]
aPlayList [1]["theme"]
aPlayList [1]["creator"]
aPlayList [1]["audience"]

# and 30+  more "properties" for the media item

OK so let's say user  want sto search "blues"  and I want the UX to check *all* 
columns in the array.  What is the best way to do this?

I tried a simple test.

on mouseup
put "apple" into aFruits["1"]
put "orange" into aFruits["2"]
put "plum" into aFruits ["3"]
put aFruits contains "orange"
end mouseup

and it returns "false"

The only other strategy I can see would be

put 1 into z # for our new filtered play list

repeat for each key x in aPlaylist
repeat for each key y in aPlayList[x]
   if aPlayList[x][y]contains "blues" then
  put aPlayList[x] into aFilteredPlayList[z]
  add 1 to z
  end if
 end repeat
end repeat

So then if we started with 1000 "records" in the array and we found "blues" in 
any column of any record, for, say 20 of them, we end up with a new array.

   aFilteredPlayList  # containing top level keys 1-20 for 20 records, put the 
titles into a list field and viola: the clickline number matches the array key 
and away we go.

OK I can make this work, (have to since I don't see another way)

But it seems like a lot of looping.

Is there a more efficient approach?

BR













___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode



___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Mike Bonner via use-livecode]

I'm with bob on this.. If for some reason you can't use the database that
produced the array, shove the data into an sqlite db instead. In memory, or
if you need persistence a db file.

Alternatively, if you store your data flat rather than in an array. Then
you can
repeat for each line tline in tData
  if tLine contains tSearchString then
   -- do whatever you want with tline
  end if
end repeat

You would still have to loop once through every record, but wouldn't have
the overhead of the subloop checking each column of each record for your
string.  It would just check the whole line.  (you could use itemoffset or
something of that nature, but its could be a pain to do so if the chosen
delimiter appears in the text. As a full string search of the line though,
it should work fine. )

On Wed, Aug 23, 2017 at 9:36 AM, Bob Sneidar via use-livecode <
use-livecode@lists.runrev.com> wrote:

> Oh wait I see you ARE using an SQL database to begin with. Why would you
> then want to search an array?
>
> Bob S
>
>
> > On Aug 23, 2017, at 08:34 , Bob Sneidar via use-livecode <
> use-livecode@lists.runrev.com> wrote:
> >
> > Yes. Don't use an array, use a memory based sqLite table.
> >
> > Bob S
> >
> >
> >> On Aug 23, 2017, at 08:24 , Sannyasin Brahmanathaswami via use-livecode
>  wrote:
> >>
> >> OK I can make this work, (have to since I don't see another way)
> >>
> >> But it seems like a lot of looping.
> >>
> >> Is there a more efficient approach?
> >>
> >> BR
> >
> >
> > ___
> > use-livecode mailing list
> > use-livecode@lists.runrev.com
> > Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> > http://lists.runrev.com/mailman/listinfo/use-livecode
>
>
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your
> subscription preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
>
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Bob Sneidar via use-livecode]

Oh wait I see you ARE using an SQL database to begin with. Why would you then 
want to search an array? 

Bob S


> On Aug 23, 2017, at 08:34 , Bob Sneidar via use-livecode 
>  wrote:
> 
> Yes. Don't use an array, use a memory based sqLite table. 
> 
> Bob S
> 
> 
>> On Aug 23, 2017, at 08:24 , Sannyasin Brahmanathaswami via use-livecode 
>>  wrote:
>> 
>> OK I can make this work, (have to since I don't see another way)
>> 
>> But it seems like a lot of looping.
>> 
>> Is there a more efficient approach?
>> 
>> BR
> 
> 
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Search Values of Array in "One Go"

[Bob Sneidar via use-livecode]

Yes. Don't use an array, use a memory based sqLite table. 

Bob S


> On Aug 23, 2017, at 08:24 , Sannyasin Brahmanathaswami via use-livecode 
>  wrote:
> 
> OK I can make this work, (have to since I don't see another way)
> 
> But it seems like a lot of looping.
> 
> Is there a more efficient approach?
> 
> BR


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode