Re: Issues with storage of data in stack

[Monte Goulding via use-livecode]

> On 24 Mar 2018, at 7:23 am, Richard Gaskin via use-livecode 
>  wrote:
> 
> Apparently he submitted a pull request last year for what seems like a very 
> nice solution, using formatting strings:
> https://github.com/livecode/livecode/pull/5433 
> 
> 
> This was per this BZ request:
> https://quality.livecode.com/show_bug.cgi?id=18150 
> 
> 
> Not sure what happened to that.  Seems like it would be a great option.

Basically that PR is something I did on a weekend that wasn’t really on the 
radar. I got interested because I was doing the AWS script library so was 
working with ISO dates. Anyway, we had some discussion internally about whether 
we should expose ICU date formatting strings rather than continue with the ones 
described by the dateFormat (which is what I did in my PR). The advantage or 
ICU strings is that many languages use them so the chance of copy/paste from 
somewhere online doing what you want is pretty high. However because the 
dateFormat exists we weren’t overly sure how to proceed there and so we moved 
on ;-)

Cheers

Monte
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Issues with storage of data in stack

[Richard Gaskin via use-livecode]

Lagi Pittas write:

> Sqlite does fine for my single user systems but needing routines to
> change between LC dates and sqlite dates is a pain.

Indeed it is, or to any format not already handled.

There are some requests in the bug DB for other formats:
https://quality.livecode.com/show_bug.cgi?id=4636
https://quality.livecode.com/show_bug.cgi?id=19601

Most interesting is Comment #4 in bz# from Monte:
https://quality.livecode.com/show_bug.cgi?id=19601#c4

Apparently he submitted a pull request last year for what seems like a 
very nice solution, using formatting strings:

https://github.com/livecode/livecode/pull/5433

This was per this BZ request:
https://quality.livecode.com/show_bug.cgi?id=18150

Not sure what happened to that.  Seems like it would be a great option.

--
 Richard Gaskin
 Fourth World Systems
 Software Design and Development for the Desktop, Mobile, and the Web
 
 ambassa...@fourthworld.comhttp://www.FourthWorld.com

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Issues with storage of data in stack

[Lagi Pittas via use-livecode]

Hi Mark,

Thanks for that - I'd forgotten I'd asked ;-)

The times you gave are super impressive - who needs mysql - Im gonna
see if I get anytime in the next month - snowed under as usual.

I think I have got it - each cluster is in effect a file on the disk?

I will understand it more when I have had time to pause the video a
few times and try to implement a very simple system although to be
honest a few thousand records should be ample in most of my stuff and
I can archive data to a history file. Sqlite does fine for my single
user systems but needing routines to change between LC dates and
sqlite dates is a pain.

Thanks again

Lagi



On 22 March 2018 at 23:28, Mark Talluto  wrote:
> Hi Lagi,
>
> Sorry about the delayed reply. I have been on a long business trip. Your
> early designs are far more sophisticated than what we put together here.
> Super impressive history you have.
>
> LiveCode really is the champion here in that we are only using arrayEncode()
> and put myArrayA into url() to store the arrays. Selecting which array
> cluster to store might be easier to understand using a video.
>
> http://canelasoftware.com/pub/canela/liveCode_array_clustering.mp4
>
> Once you understand how the array is structured, I think the method will be
> clear.
>
> We do not preallocate space. No appending. We overwrite a cluster when one
> or more records are saved to disk. The write happens at the end of the CRUD
> operation taking place. Thus, if you ‘create’ a single record, the record is
> first created in memory and then the cluster it belongs to is written to
> disk. I have toyed with the idea of making the write to disk feature
> controllable by the dev. Thus, you could define when the write is to take
> place. For example, you might like to write to disk after every 5
> transactions or so. But, I have not found the write to affect performance in
> a noticeable way to need to add that feature.
>
> -Multi User-
> Yes, everything is processed sequentially in the cloud. There are no open
> sockets so you can have massive concurrent connections. All cloud calls are
> done via ‘post’. They are handled by PHP scripts to write the request to a
> cache area. One or more LiveCode standalones on the other end processes the
> request in the order they are received. Thus, should a process go down, no
> data is lost. When the process comes a back up, everything continues again
> as normal. Scale is handled by having more than one process be available.
> More scaling is handled by having data stored across multiple droplets/VMs
> (sharding). This can keep repeating itself as needed.
>
> -File Size Limitations-
> The OS iNode limitations are negated by not reaching its maximum limit. We
> found 40,000 files would really bring the performance down. Adding
> clustering of arrays lowers the file count to acceptable and controllable
> levels.
>
> -Test Data-
> 100,000 records in table
> Record size average: 45 chars
> Keys in each record: last_name, first_name, date_of_birth, middle_name,
> student_number, gender, grade_level, active
>
> A cluster size, clusters per table, time to load all clusters from disk to
> RAM, time to write all clusters from RAM to disk, time to write one cluster
> from RAM to disk:
> 1, 16, 1.46 secs, 1.5 secs, 91.4 ms
> 2, 256, 1.52 secs, 1.5 secs, 6.7 ms
> 3, 4096, 2.38 secs, 1.6 secs, 0.8 ms
>
> I hope this information is helpful. Please let me know if you have any other
> questions.
>
> Best regards,
>
> Mark Talluto
> livecloud.io
> nursenotes.net
> canelasoftware.com
>
>
>
>
> On Mar 12, 2018, at 10:31 AM, Lagi Pittas  wrote:
>
> Hi Mark,
>
> Thanks for the detailed explanation but I have a few (ish) questions ...
>
> Hope you don't mind me asking these questions because I did have to
> write my own access routines in those bad old days before I started on
> Clipper/Foxpro/Delphi/Btrieve  and I do enjoy learning from others on
> the list and the forums - those AHA! moments when you finally get how
> the Heapsort works the night before the exam.
>
> Many moons ago I wrote a multi-way B-TREE based  on the explanation in
> Wirth's Book "Algorithms + Data Structures = Programs" -  in UCSD
> Pascal for the Apple 2,  I  had a 5MB hard Drives for the bigger
> companies when I was lucky, for the smaller companies I made do with 2
> 143k floppy disks and Hashing for a "large" data set- oh the memories.
> I used   the B-Trees  if the codes were alphanumeric. I also had my
> own method where I kept the index in the first X Blocks of the file
> and loaded the parts in memory as they were needed - a brain dead
> version of yours I suppose.  I think we had about 40k of free ram to
> Play with so couldn't always keep everything in RAM. I even made the
> system multi-user and ran 20 Apple ][s on a network using a
> proprietary Nestar/Zynar network using Ribbon Cables -  it worked but
> am I glad we have Ethernet!
>
> Anyway - I digress. I can understand the general idea of what you are
> explaining but it's t

Re: Issues with storage of data in stack

[Mark Talluto via use-livecode]

Hi Lagi,

Sorry about the delayed reply. I have been on a long business trip. Your early 
designs are far more sophisticated than what we put together here. Super 
impressive history you have.

LiveCode really is the champion here in that we are only using arrayEncode() 
and put myArrayA into url() to store the arrays. Selecting which array cluster 
to store might be easier to understand using a video. 

http://canelasoftware.com/pub/canela/liveCode_array_clustering.mp4

Once you understand how the array is structured, I think the method will be 
clear.

We do not preallocate space. No appending. We overwrite a cluster when one or 
more records are saved to disk. The write happens at the end of the CRUD 
operation taking place. Thus, if you ‘create’ a single record, the record is 
first created in memory and then the cluster it belongs to is written to disk. 
I have toyed with the idea of making the write to disk feature controllable by 
the dev. Thus, you could define when the write is to take place. For example, 
you might like to write to disk after every 5 transactions or so. But, I have 
not found the write to affect performance in a noticeable way to need to add 
that feature.

-Multi User-
Yes, everything is processed sequentially in the cloud. There are no open 
sockets so you can have massive concurrent connections. All cloud calls are 
done via ‘post’. They are handled by PHP scripts to write the request to a 
cache area. One or more LiveCode standalones on the other end processes the 
request in the order they are received. Thus, should a process go down, no data 
is lost. When the process comes a back up, everything continues again as 
normal. Scale is handled by having more than one process be available. More 
scaling is handled by having data stored across multiple droplets/VMs 
(sharding). This can keep repeating itself as needed.

-File Size Limitations-
The OS iNode limitations are negated by not reaching its maximum limit. We 
found 40,000 files would really bring the performance down. Adding clustering 
of arrays lowers the file count to acceptable and controllable levels. 

-Test Data-
100,000 records in table
Record size average: 45 chars
Keys in each record: last_name, first_name, date_of_birth, middle_name, 
student_number, gender, grade_level, active

A cluster size, clusters per table, time to load all clusters from disk to RAM, 
time to write all clusters from RAM to disk, time to write one cluster from RAM 
to disk:
1, 16, 1.46 secs, 1.5 secs, 91.4 ms
2, 256, 1.52 secs, 1.5 secs, 6.7 ms
3, 4096, 2.38 secs, 1.6 secs, 0.8 ms

I hope this information is helpful. Please let me know if you have any other 
questions.

Best regards,

Mark Talluto
livecloud.io 
nursenotes.net 
canelasoftware.com 




> On Mar 12, 2018, at 10:31 AM, Lagi Pittas  wrote:
> 
> Hi Mark,
> 
> Thanks for the detailed explanation but I have a few (ish) questions ...
> 
> Hope you don't mind me asking these questions because I did have to
> write my own access routines in those bad old days before I started on
> Clipper/Foxpro/Delphi/Btrieve  and I do enjoy learning from others on
> the list and the forums - those AHA! moments when you finally get how
> the Heapsort works the night before the exam.
> 
> Many moons ago I wrote a multi-way B-TREE based  on the explanation in
> Wirth's Book "Algorithms + Data Structures = Programs" -  in UCSD
> Pascal for the Apple 2,  I  had a 5MB hard Drives for the bigger
> companies when I was lucky, for the smaller companies I made do with 2
> 143k floppy disks and Hashing for a "large" data set- oh the memories.
> I used   the B-Trees  if the codes were alphanumeric. I also had my
> own method where I kept the index in the first X Blocks of the file
> and loaded the parts in memory as they were needed - a brain dead
> version of yours I suppose.  I think we had about 40k of free ram to
> Play with so couldn't always keep everything in RAM. I even made the
> system multi-user and ran 20 Apple ][s on a network using a
> proprietary Nestar/Zynar network using Ribbon Cables -  it worked but
> am I glad we have Ethernet!
> 
> Anyway - I digress. I can understand the general idea of what you are
> explaining but it's the low level code for writing to the
> clusters/file on disk I'm not quite sure of.
> Which way do you build your initial file? Is it "Sparse" or prebuilt,
> or does each cluster  have a "pointer" to previous or next clusters?
> Do you have records "spanning" clusters or do you leave any spare
> space in a cluster empty. Do you mark a "record" as deleted but don't
> remove the record until it's overwritten or do what Foxpro/Dbase does
> and "PACK" them with a utility routine.
> I also presume you use the "AT" option in the write command to write
> the clusters randomly since you don't wriite the whole in memory table
> 
> Which brings me onto my final questions - I presume your system is
> mu

Re: Issues with storage of data in stack

[Lagi Pittas via use-livecode]

Hi Mark,

Thanks for the detailed explanation but I have a few (ish) questions ...

Hope you don't mind me asking these questions because I did have to
write my own access routines in those bad old days before I started on
Clipper/Foxpro/Delphi/Btrieve  and I do enjoy learning from others on
the list and the forums - those AHA! moments when you finally get how
the Heapsort works the night before the exam.

Many moons ago I wrote a multi-way B-TREE based  on the explanation in
Wirth's Book "Algorithms + Data Structures = Programs" -  in UCSD
Pascal for the Apple 2,  I  had a 5MB hard Drives for the bigger
companies when I was lucky, for the smaller companies I made do with 2
143k floppy disks and Hashing for a "large" data set- oh the memories.
I used   the B-Trees  if the codes were alphanumeric. I also had my
own method where I kept the index in the first X Blocks of the file
and loaded the parts in memory as they were needed - a brain dead
version of yours I suppose.  I think we had about 40k of free ram to
Play with so couldn't always keep everything in RAM. I even made the
system multi-user and ran 20 Apple ][s on a network using a
proprietary Nestar/Zynar network using Ribbon Cables -  it worked but
am I glad we have Ethernet!

Anyway - I digress. I can understand the general idea of what you are
explaining but it's the low level code for writing to the
clusters/file on disk I'm not quite sure of.
Which way do you build your initial file? Is it "Sparse" or prebuilt,
or does each cluster  have a "pointer" to previous or next clusters?
Do you have records "spanning" clusters or do you leave any spare
space in a cluster empty. Do you mark a "record" as deleted but don't
remove the record until it's overwritten or do what Foxpro/Dbase does
and "PACK" them with a utility routine.
I also presume you use the "AT" option in the write command to write
the clusters randomly since you don't wriite the whole in memory table

Which brings me onto my final questions - I presume your system is
multi-user because you have a server program that receives calls and
executes them sequentially? And lastly what are the file size
limitations doing it this way - do You also virtualize the data in
memory?

Sorry for all the question but this is the interesting stuff

Regards Lagi

On 11 March 2018 at 20:02, Mark Talluto  wrote:
> Hi Lagi,
>
> Our LiveCode array database does not use SQL or any other database. In terms
> of local only database, we do not rely on anything but LiveCode. It is
> purely a LiveCode derived system. Data is manipulated using familiar
> methodologies to other databases. The data is encrypted using ‘encrypt’ and
> stored using 'arrayEncode()’.
>
> The full array of the database is stored in memory. This method provides
> very quick access to your data thanks to the amazing performance provided by
> LiveCode.
>
> This might get a little long. I am happy to take this off list for more
> details. I will try to be as succinct as possible.
>
> -A little more explanation on storing data-
> Each record is stored in array that looks like this:
> tableID/clusterID/recordID/recordData…
> When a given record/s is updated, we cache the recordIDs that were touched.
> All data is updated first in memory then cached to disk. We then refer to
> the cached records and conclude which clusterIDs we affected. Thus, you can
> very quickly save only the clusters that have been modified. Each cluster
> will have one or more records associated with it. The clusters are the first
> ’n’ characters of the recordIDs. We use UUIDs as recordIDs. The cluster
> sizes are definable, giving us the power to decide where to apply the
> optimization. Clusters of only one or two characters will generate less
> clusters to be stored. This makes loading of tables from disk to RAM very
> fast. A cluster of 3 chars or more allows for less records per cluster thus
> improving saving from RAM to disk to be faster.
>
> The pseudo code for this might looks like this:
> -receive request for update in your API
> -store the changes to your master array in RAM
> -remember the recordIDs touched in a variable
> -calculate the clusters touched by taking the first ’n’ characters of the
> records touched and make a new list of the clusters you need to write to
> disk
> -write appropriate clusters to disk
> -return the results of the action (any errors, recordIDs…)
>
> You will find this method to be very performant and easy to manage. This is
> not particularly complicated to write. Once you get it all working you might
> add other niceties like:
> -error checking the input before storing anything
> -store metadata on each updated record: recordVersion, recordTime,
> updateTime
> -add security using ‘encrypt’
> -build simple APIs to do your CRUD first
> -add other APIs as needed to make accessing your data easier
>
> Here is an example API for storing data that you may find useful when making
> your own system.
>
> -Input (array)-
> put “Steve” into tInput

Re: Issues with storage of data in stack

[Mark Talluto via use-livecode]

Hi BR,

Sorry about that. I have updated the docs page to link back to the main site.  
https://livecloud.io 

Mark


> On Mar 11, 2018, at 7:18 PM, Sannyasin Brahmanathaswami via use-livecode 
>  wrote:
> 
> Mark Talluto wrote:
> 
>If you are interested in seeing other APIs we have found useful, here is a 
> running list. Look under the API dropdown for more ideas.
>http://canelasoftware.github.io/cassiaDB_docs/ 
> http://canelasoftware.github.io/cassiaDB_docs/
> 
> Where does one download the LiveCloud Manager application (duh, I am not 
> seeing something) 
> 
> BR
> 
> 
> 
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Issues with storage of data in stack

[Sannyasin Brahmanathaswami via use-livecode]

Mark Talluto wrote:

If you are interested in seeing other APIs we have found useful, here is a 
running list. Look under the API dropdown for more ideas.
http://canelasoftware.github.io/cassiaDB_docs/ 
http://canelasoftware.github.io/cassiaDB_docs/

Where does one download the LiveCloud Manager application (duh, I am not seeing 
something) 

BR



___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Issues with storage of data in stack

[Mark Talluto via use-livecode]

Hi Lagi,

Our LiveCode array database does not use SQL or any other database. In terms of 
local only database, we do not rely on anything but LiveCode. It is purely a 
LiveCode derived system. Data is manipulated using familiar methodologies to 
other databases. The data is encrypted using ‘encrypt’ and stored using 
'arrayEncode()’.  

The full array of the database is stored in memory. This method provides very 
quick access to your data thanks to the amazing performance provided by 
LiveCode. 

This might get a little long. I am happy to take this off list for more 
details. I will try to be as succinct as possible.

-A little more explanation on storing data-
Each record is stored in array that looks like this: 
tableID/clusterID/recordID/recordData…
When a given record/s is updated, we cache the recordIDs that were touched. All 
data is updated first in memory then cached to disk. We then refer to the 
cached records and conclude which clusterIDs we affected. Thus, you can very 
quickly save only the clusters that have been modified. Each cluster will have 
one or more records associated with it. The clusters are the first ’n’ 
characters of the recordIDs. We use UUIDs as recordIDs. The cluster sizes are 
definable, giving us the power to decide where to apply the optimization. 
Clusters of only one or two characters will generate less clusters to be 
stored. This makes loading of tables from disk to RAM very fast. A cluster of 3 
chars or more allows for less records per cluster thus improving saving from 
RAM to disk to be faster.

The pseudo code for this might looks like this:
-receive request for update in your API
-store the changes to your master array in RAM
-remember the recordIDs touched in a variable
-calculate the clusters touched by taking the first ’n’ characters of the 
records touched and make a new list of the clusters you need to write to disk
-write appropriate clusters to disk
-return the results of the action (any errors, recordIDs…)

You will find this method to be very performant and easy to manage. This is not 
particularly complicated to write. Once you get it all working you might add 
other niceties like:
-error checking the input before storing anything
-store metadata on each updated record: recordVersion, recordTime, updateTime
-add security using ‘encrypt’
-build simple APIs to do your CRUD first
-add other APIs as needed to make accessing your data easier

Here is an example API for storing data that you may find useful when making 
your own system.

-Input (array)-
put “Steve” into tInputA[“firstName”]
put “Jobs” into tInputA[“lastName”]
put “rolodex” into tInputA[“cdbTableName”]
put “local” into tInputA[“cdbTarget”] —We would use ‘cloud’ when we want to 
store offsite.

Your system might verify that the keys ‘firstName' and ‘lastName' are actual 
keys. This is very SQL in terms of error checking. Or, do not check the keys 
and feel more noSQL in nature.

From here we take the array and pass it to a function. 
put cdb_create(tInputA) into tRecordID —returns the unique UUID representing 
the recordID

The ‘cdb_create()’ function basically runs the pseudo code proposed above. And 
voilà, you have your first record stored.

I hope this gives you some ideas. We have successfully used this in enterprise 
level scenarios. We nightly interchange data with our customer's database. We 
have not run into any issues with IT in terms of the type of database we are 
using in our backend. I fully support your experimentation with arrays as a 
data store.  :)

If you are interested in seeing other APIs we have found useful, here is a 
running list. Look under the API dropdown for more ideas.
http://canelasoftware.github.io/cassiaDB_docs/ 


-Mark


> On Mar 9, 2018, at 2:07 AM, Lagi Pittas  wrote:
> 
> HI Mark,
> 
> I am intrigued by your way of saving only whats changed and also when
> you say save to disk after arrayencoded. Do you meanas a simple  save
> as a binfile or in an sqlite BLOB?
> 
> I would really like to see some exampleish code on saving to disk - if
> it's other than a single array in a single binfile - I think even I
> can do that. But your other ideas sound brilliant.
> 
> Regards Lagi
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Issues with storage of data in stack

[Richard Gaskin via use-livecode]

Some notes from a while back on using LSON (LiveCode encoded arrays):

http://lists.runrev.com/pipermail/use-livecode/2016-May/227330.html

--
 Richard Gaskin
 Fourth World Systems
 Software Design and Development for the Desktop, Mobile, and the Web
 
 ambassa...@fourthworld.comhttp://www.FourthWorld.com

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Issues with storage of data in stack

[Lagi Pittas via use-livecode]

HI Mark,

I am intrigued by your way of saving only whats changed and also when
you say save to disk after arrayencoded. Do you meanas a simple  save
as a binfile or in an sqlite BLOB?

I would really like to see some exampleish code on saving to disk - if
it's other than a single array in a single binfile - I think even I
can do that. But your other ideas sound brilliant.

Regards Lagi

On 8 March 2018 at 18:38, Mark Talluto via use-livecode
 wrote:
>> On Mar 8, 2018, at 9:03 AM, Lagi Pittas via use-livecode 
>>  wrote:
>>
>> In the first version of the system I save the arrays which held all
>> the Clerks orders/totals etc and items into an external stack every
>> time they added an item.
>
> I too use arrays as a data store. You could save the arrays to disk directly 
> with arrayEncode(). To save time and not store the entire array to disk, you 
> could store only the portions that have been updated on every update. This 
> will keep the data store safe and fast. It will also remove the occasional 
> issue of stack corruption…since you will not store the data inside of stacks 
> any longer.
>
> All creates/reads/update/writes would be done to memory. Those calls that 
> modify data will simply do a follow-up write to disk for that particular 
> record. To defeat OS inode limitations, you could group your arrays logically 
> into files. Thus, you would have anywhere from 1 to a 100 sub arrays grouped 
> together. The loading and storing of arrays is very fast. Even with databases 
> in the 100s of thousands to low millions of records counts.
>
> Performance is amazing. The best part is you are speaking to your database 
> using LiveCode commands and functions. The only limitation is the amount of 
> RAM on a given device. Should you outgrow that limitation then you could move 
> the data to the cloud. Or you could load/unload data accordingly from disk. 
> This would also be a nice indicator that your business is growing. Of course, 
> it could just be that you are storing a lot of data. :)
>
> If you need any help or ideas, let me know.
>
> Best regards,
>
> Mark Talluto
> livecloud.io 
> nursenotes.net 
> canelasoftware.com 
>
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Issues with storage of data in stack

[Mark Talluto via use-livecode]

Hi Peter,

Yes. But, you can protect it with a password. 

1. arrayEncode(yourDataInAVariable)
2. Encrypt the result above
3. Write out the encrypted value to disk

Best regards,

Mark Talluto
livecloud.io 
nursenotes.net 
canelasoftware.com 


> On Mar 8, 2018, at 11:22 AM, Peter Bogdanoff via use-livecode 
>  wrote:
> 
> Mark,
> 
> If the data is written to a file on disk, could someone else arrayDecode the 
> file and gain access to it?
> 
> Peter Bogdanoff
> ArtsInteractive
> 
>> On Mar 8, 2018, at 10:38 AM, Mark Talluto via use-livecode 
>>  wrote:
>> 
>>> On Mar 8, 2018, at 9:03 AM, Lagi Pittas via use-livecode 
>>>  wrote:
>>> 
>>> In the first version of the system I save the arrays which held all
>>> the Clerks orders/totals etc and items into an external stack every
>>> time they added an item.
>> 
>> I too use arrays as a data store. You could save the arrays to disk directly 
>> with arrayEncode(). To save time and not store the entire array to disk, you 
>> could store only the portions that have been updated on every update. This 
>> will keep the data store safe and fast. It will also remove the occasional 
>> issue of stack corruption…since you will not store the data inside of stacks 
>> any longer.
>> 
>> All creates/reads/update/writes would be done to memory. Those calls that 
>> modify data will simply do a follow-up write to disk for that particular 
>> record. To defeat OS inode limitations, you could group your arrays 
>> logically into files. Thus, you would have anywhere from 1 to a 100 sub 
>> arrays grouped together. The loading and storing of arrays is very fast. 
>> Even with databases in the 100s of thousands to low millions of records 
>> counts. 
>> 
>> Performance is amazing. The best part is you are speaking to your database 
>> using LiveCode commands and functions. The only limitation is the amount of 
>> RAM on a given device. Should you outgrow that limitation then you could 
>> move the data to the cloud. Or you could load/unload data accordingly from 
>> disk. This would also be a nice indicator that your business is growing. Of 
>> course, it could just be that you are storing a lot of data. :)
>> 
>> If you need any help or ideas, let me know.
>> 
>> Best regards,
>> 
>> Mark Talluto
>> livecloud.io 
>> nursenotes.net 
>> canelasoftware.com 
>> 
>> ___
>> use-livecode mailing list
>> use-livecode@lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your subscription 
>> preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
> 
> 
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Issues with storage of data in stack

[Phil Davis via use-livecode]

Hi Peter,

If your app could encrypt the arrayEncoded data before writing to disk, 
and then decrypt it when reading it, that would solve the 
unwanted-access problem. The amount of time used by encrypt/decrypt 
processes is negligible, even for large files.


I love arrays. They have become one of my favorite things about LC.

Phil Davis


On 3/8/18 11:22 AM, Peter Bogdanoff via use-livecode wrote:

Mark,

If the data is written to a file on disk, could someone else arrayDecode the 
file and gain access to it?

Peter Bogdanoff
ArtsInteractive


On Mar 8, 2018, at 10:38 AM, Mark Talluto via use-livecode 
 wrote:


On Mar 8, 2018, at 9:03 AM, Lagi Pittas via use-livecode 
 wrote:

In the first version of the system I save the arrays which held all
the Clerks orders/totals etc and items into an external stack every
time they added an item.

I too use arrays as a data store. You could save the arrays to disk directly 
with arrayEncode(). To save time and not store the entire array to disk, you 
could store only the portions that have been updated on every update. This will 
keep the data store safe and fast. It will also remove the occasional issue of 
stack corruption…since you will not store the data inside of stacks any longer.

All creates/reads/update/writes would be done to memory. Those calls that 
modify data will simply do a follow-up write to disk for that particular 
record. To defeat OS inode limitations, you could group your arrays logically 
into files. Thus, you would have anywhere from 1 to a 100 sub arrays grouped 
together. The loading and storing of arrays is very fast. Even with databases 
in the 100s of thousands to low millions of records counts.

Performance is amazing. The best part is you are speaking to your database 
using LiveCode commands and functions. The only limitation is the amount of RAM 
on a given device. Should you outgrow that limitation then you could move the 
data to the cloud. Or you could load/unload data accordingly from disk. This 
would also be a nice indicator that your business is growing. Of course, it 
could just be that you are storing a lot of data. :)

If you need any help or ideas, let me know.

Best regards,

Mark Talluto
livecloud.io 
nursenotes.net 
canelasoftware.com 

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode


--
Phil Davis


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Issues with storage of data in stack

[Bob Sneidar via use-livecode]

Yes, if they had Livecode. In that case encrypt the arraydecode string using a 
seed that only you know. I use a special seed value that is different for every 
record I add to a database. Very effective and hard to guess. 

Bob S


> On Mar 8, 2018, at 11:22 , Peter Bogdanoff via use-livecode 
>  wrote:
> 
> Mark,
> 
> If the data is written to a file on disk, could someone else arrayDecode the 
> file and gain access to it?
> 
> Peter Bogdanoff


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Issues with storage of data in stack

[Peter Bogdanoff via use-livecode]

Mark,

If the data is written to a file on disk, could someone else arrayDecode the 
file and gain access to it?

Peter Bogdanoff
ArtsInteractive

> On Mar 8, 2018, at 10:38 AM, Mark Talluto via use-livecode 
>  wrote:
> 
>> On Mar 8, 2018, at 9:03 AM, Lagi Pittas via use-livecode 
>>  wrote:
>> 
>> In the first version of the system I save the arrays which held all
>> the Clerks orders/totals etc and items into an external stack every
>> time they added an item.
> 
> I too use arrays as a data store. You could save the arrays to disk directly 
> with arrayEncode(). To save time and not store the entire array to disk, you 
> could store only the portions that have been updated on every update. This 
> will keep the data store safe and fast. It will also remove the occasional 
> issue of stack corruption…since you will not store the data inside of stacks 
> any longer.
> 
> All creates/reads/update/writes would be done to memory. Those calls that 
> modify data will simply do a follow-up write to disk for that particular 
> record. To defeat OS inode limitations, you could group your arrays logically 
> into files. Thus, you would have anywhere from 1 to a 100 sub arrays grouped 
> together. The loading and storing of arrays is very fast. Even with databases 
> in the 100s of thousands to low millions of records counts. 
> 
> Performance is amazing. The best part is you are speaking to your database 
> using LiveCode commands and functions. The only limitation is the amount of 
> RAM on a given device. Should you outgrow that limitation then you could move 
> the data to the cloud. Or you could load/unload data accordingly from disk. 
> This would also be a nice indicator that your business is growing. Of course, 
> it could just be that you are storing a lot of data. :)
> 
> If you need any help or ideas, let me know.
> 
> Best regards,
> 
> Mark Talluto
> livecloud.io 
> nursenotes.net 
> canelasoftware.com 
> 
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Issues with storage of data in stack

[Bob Sneidar via use-livecode]

You would need to arrayencode() before saving an array to a database as well. 

Bob S


> On Mar 8, 2018, at 10:38 , Mark Talluto via use-livecode 
>  wrote:
> 
> I too use arrays as a data store. You could save the arrays to disk directly 
> with arrayEncode(). 


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Issues with storage of data in stack

[Mark Talluto via use-livecode]

> On Mar 8, 2018, at 9:03 AM, Lagi Pittas via use-livecode 
>  wrote:
> 
> In the first version of the system I save the arrays which held all
> the Clerks orders/totals etc and items into an external stack every
> time they added an item.

I too use arrays as a data store. You could save the arrays to disk directly 
with arrayEncode(). To save time and not store the entire array to disk, you 
could store only the portions that have been updated on every update. This will 
keep the data store safe and fast. It will also remove the occasional issue of 
stack corruption…since you will not store the data inside of stacks any longer.

All creates/reads/update/writes would be done to memory. Those calls that 
modify data will simply do a follow-up write to disk for that particular 
record. To defeat OS inode limitations, you could group your arrays logically 
into files. Thus, you would have anywhere from 1 to a 100 sub arrays grouped 
together. The loading and storing of arrays is very fast. Even with databases 
in the 100s of thousands to low millions of records counts. 

Performance is amazing. The best part is you are speaking to your database 
using LiveCode commands and functions. The only limitation is the amount of RAM 
on a given device. Should you outgrow that limitation then you could move the 
data to the cloud. Or you could load/unload data accordingly from disk. This 
would also be a nice indicator that your business is growing. Of course, it 
could just be that you are storing a lot of data. :)

If you need any help or ideas, let me know.

Best regards,

Mark Talluto
livecloud.io 
nursenotes.net 
canelasoftware.com 

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Issues with storage of data in stack

[J. Landman Gay via use-livecode]

I've found that sending binary data over the internet is risky. A zip file 
or base64 encoded file avoids most transit corruption.

--
Jacqueline Landman Gay | jac...@hyperactivesw.com
HyperActive Software   | http://www.hyperactivesw.com



On March 8, 2018 11:06:16 AM Lagi Pittas via use-livecode 
 wrote:



Hi Mike,

I have 60+ machines running windows7  embedded  running a POS/TILL
system also a few  tablets running windows and android with the same
system - wth abouth 8 lines of code added to work on Android.

 In the first version of the system I save the arrays which held all
the Clerks orders/totals etc and items into an external stack every
time they added an item.

I then saved the whole thing and cleared the  "dirty" flag   on shutdown.

If the"till"  program started and the dirty flag was true then system
would know it was a powercut and load all the "open"  orders and other
status information  etc.

I have users who have not had a problem in over 2 and a half  years
others who have 3 "tills" with no problems and just a few maybe (4 who
have had a corrupt stack maybe 1 to 3 times a years - 2 or three times
is  too many. There is no rhyme or reason because others have had
powercuts (6 in an hour ) with no damage but others can can switch off
without exiting the till properly without a problem and yet I can have
a corrupt stack with everything shut down properly.  So Now I have
removed that stack and  just save the arrays to  "general" config
table called TABS in sqlite in a row with an idkey of '99' other
IDkeys can save other configuration information- NOT ONE PROBLEM since
I have done this - go figure.

I'll make a posting on the forum with the code - not trying to teach
my granny to suck eggs here, but maybe some beginners on the forum
might find it useful - would have done in the beginning.


http://forums.livecode.com/viewtopic.php?f=7&t=30717

Regards lagi

On 7 March 2018 at 20:30, Mike Kerner via use-livecode
 wrote:

I'm working on a session for next week's LCG.  We are all working on
fun super-advanced pushing-the-envelope things in LC, but I wanted to do
something for the beginners who are watching LCG, and handling data seemed
like an obvious choice.

I have to admit, I haven't stored data in a stack in a long time, since the
database options are so easy to use.  If you're currently storing data in a
stack, have you run into any issues on any platform with it, especially on
mobile, for instance, where the rules can be a little weird?

--
On the first day, God created the heavens and the Earth
On the second day, God created the oceans.
On the third day, God put the animals on hold for a few hours,
   and did a little diving.
And God said, "This is good."
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your 
subscription preferences:

http://lists.runrev.com/mailman/listinfo/use-livecode


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your 
subscription preferences:

http://lists.runrev.com/mailman/listinfo/use-livecode




___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Issues with storage of data in stack

[Lagi Pittas via use-livecode]

Hi Mike,

I have 60+ machines running windows7  embedded  running a POS/TILL
system also a few  tablets running windows and android with the same
system - wth abouth 8 lines of code added to work on Android.

 In the first version of the system I save the arrays which held all
the Clerks orders/totals etc and items into an external stack every
time they added an item.

I then saved the whole thing and cleared the  "dirty" flag   on shutdown.

If the"till"  program started and the dirty flag was true then system
would know it was a powercut and load all the "open"  orders and other
status information  etc.

I have users who have not had a problem in over 2 and a half  years
others who have 3 "tills" with no problems and just a few maybe (4 who
have had a corrupt stack maybe 1 to 3 times a years - 2 or three times
is  too many. There is no rhyme or reason because others have had
powercuts (6 in an hour ) with no damage but others can can switch off
without exiting the till properly without a problem and yet I can have
a corrupt stack with everything shut down properly.  So Now I have
removed that stack and  just save the arrays to  "general" config
table called TABS in sqlite in a row with an idkey of '99' other
IDkeys can save other configuration information- NOT ONE PROBLEM since
I have done this - go figure.

I'll make a posting on the forum with the code - not trying to teach
my granny to suck eggs here, but maybe some beginners on the forum
might find it useful - would have done in the beginning.


http://forums.livecode.com/viewtopic.php?f=7&t=30717

Regards lagi

On 7 March 2018 at 20:30, Mike Kerner via use-livecode
 wrote:
> I'm working on a session for next week's LCG.  We are all working on
> fun super-advanced pushing-the-envelope things in LC, but I wanted to do
> something for the beginners who are watching LCG, and handling data seemed
> like an obvious choice.
>
> I have to admit, I haven't stored data in a stack in a long time, since the
> database options are so easy to use.  If you're currently storing data in a
> stack, have you run into any issues on any platform with it, especially on
> mobile, for instance, where the rules can be a little weird?
>
> --
> On the first day, God created the heavens and the Earth
> On the second day, God created the oceans.
> On the third day, God put the animals on hold for a few hours,
>and did a little diving.
> And God said, "This is good."
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Issues with storage of data in stack

[dunbarx via use-livecode]

Hi.

I store data in both external files and within the stack itself. Never used
a DB.

But no issues of any kind ever.

Craig Newman



--
Sent from: 
http://runtime-revolution.278305.n4.nabble.com/Revolution-User-f278306.html

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Issues with storage of data in stack

[Richard Gaskin via use-livecode]

Mike Kerner wrote:

> I have to admit, I haven't stored data in a stack in a long time,
> since the database options are so easy to use.  If you're currently
> storing data in a stack, have you run into any issues on any platform
> with it, especially on mobile, for instance, where the rules can be a
> little weird?

In fields, or custom props?

--
 Richard Gaskin
 Fourth World Systems
 Software Design and Development for the Desktop, Mobile, and the Web
 
 ambassa...@fourthworld.comhttp://www.FourthWorld.com

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode