Re: Divide Large Data Blob?

[Richard Gaskin via use-livecode]

There may be other approaches, but without knowing what the data looks 
like, what the resulting index should look like, and how big the data is 
we'd just be guessing.


--
 Richard Gaskin
 Fourth World Systems



Rick Harrison wrote:

Hi Richard,

I was looking at the offset function and thinking about
the starting points.  It still presents a looping problem
for me that I’m trying to avoid.  If other methods aren’t
more efficient I will play with it more.

Thanks,

Rick


On May 16, 2022, at 2:32 PM, Richard Gaskin via use-livecode  wrote:

The offset function has an optional third param for starting point, so setting 
that value to the number of bytes you want to skip would take you right where 
you want to be. Offset can then be used in a loop, updating the starting value 
as you go. Exiting the loop once your into the tail you don't want to index can 
be a simple if to escape the loop.

There may also be options for using delimiters. What does the source data look 
like, and what should the resulting index look like?

And the most central question with things like this: how large is "large"?





___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Divide Large Data Blob?

[Rick Harrison via use-livecode]

Hi Bob,

Nice to know this for the future.

Thanks for your research!

Rick

> On May 16, 2022, at 7:53 PM, Bob Sneidar via use-livecode 
>  wrote:
> 
> OK so it appears there is a log2 property. The log2 of 1000 yields 9.965784 
> so I suppsoe if you round up, that would give you the maximum number of 
> iterations to isolate a single line in a 1000 line sorted list. 
> 
> Bob S

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Divide Large Data Blob?

[Rick Harrison via use-livecode]

Hi Bob,

Yes, I was looking into the binary sort idea.  It’s too bad that
isn’t just a built-in function for LC.  I’m not sure it’s worth
that kind of effort for my particular case.

Thanks,

Rick



> On May 16, 2022, at 6:44 PM, Bob Sneidar via use-livecode 
>  wrote:
> 
> So this has got me thinking. Apparently what I am calling Divide and Conquer 
> is really called a binary sort. I have looked up on the interwebs to 
> calculate the maximum number of iterations for a given number of values, but 
> it seems that all the formulas offered up use functions for C. I am trying to 
> figure out what a basic math formula for this is, given n values. 
> 
> Bob S
> 


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Divide Large Data Blob?

[Rick Harrison via use-livecode]

Hi Bob,

I just need to make one pass at the data, so building an SQL database for it 
doesn’t make sense.

I don’t have any definitive line for the boundary between good and bad data.  
It’s more of a
consistent first guess used just to cut down on the amount of data to process.  
I start looking
for the good data after about a third of it has passed by.

Thanks,

Rick

> On May 16, 2022, at 6:00 PM, Bob Sneidar via use-livecode 
>  wrote:
> 
> Do you know exactly which lines you need to toss, or do you need to searc the 
> data to find out where the beginning and end of the useful data is? 
> If the former, then just put line x to y of your data into a new variable. If 
> the latter, then a divide and conquer approach might be the answer. Get the 
> line 30% in, test for valid, get the line 40% in, test, then 35% then 32.5% 
> or 37.5% depending on your test. 
> 
> You may only have to do this a dozen or so times to find the exact line where 
> your valid data begins. 
> 
> The other way of course is to get it all into a SQL database (how did you all 
> know I was going to say that??) The downside is that you have to iterate 
> through all your data once. The upside is a good one liner query statement 
> may be all you need to process your data. And if you need to make multiple 
> passes at your data, all the better. 
> 
> Bob S


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Divide Large Data Blob?

[Bob Sneidar via use-livecode]

OK so it appears there is a log2 property. The log2 of 1000 yields 9.965784 so 
I suppsoe if you round up, that would give you the maximum number of iterations 
to isolate a single line in a 1000 line sorted list. 

Bob S


> On May 16, 2022, at 15:44 , Bob Sneidar via use-livecode 
>  wrote:
> 
> So this has got me thinking. Apparently what I am calling Divide and Conquer 
> is really called a binary sort. I have looked up on the interwebs to 
> calculate the maximum number of iterations for a given number of values, but 
> it seems that all the formulas offered up use functions for C. I am trying to 
> figure out what a basic math formula for this is, given n values. 
> 
> Bob S
> 
> 
>> On May 16, 2022, at 15:23 , Bob Sneidar via use-livecode 
>>  wrote:
>> 
>> A maximum of 7 recursions are necessary to isolate a single instance of 100 
>> possible values. 1000 requires a maximum of 10. 1 values requires 14. 
>> The idea is that for every factor of 10, you need roughly 3 more recursions. 
>> This of course assumes the data is sorted, which in your case is sorted into 
>> 3 containers. If you know the limits of how many lines can be garbage, and 
>> how many can be valid data, you narrow your scope significantly. 
>> 
>> Livecode is pretty damn quick at parsing this kind of data. If there are 
>> consistent delimiters (in this case a line break) then even 20 or 30 
>> recursions is child's play. 
>> 
>> Bob S
>> 
>> 
>>> On May 16, 2022, at 15:00 , Bob Sneidar via use-livecode 
>>>  wrote:
>>> 
>>> Do you know exactly which lines you need to toss, or do you need to searc 
>>> the data to find out where the beginning and end of the useful data is? 
>>> If the former, then just put line x to y of your data into a new variable. 
>>> If the latter, then a divide and conquer approach might be the answer. Get 
>>> the line 30% in, test for valid, get the line 40% in, test, then 35% then 
>>> 32.5% or 37.5% depending on your test. 
>>> 
>>> You may only have to do this a dozen or so times to find the exact line 
>>> where your valid data begins. 
>>> 
>>> The other way of course is to get it all into a SQL database (how did you 
>>> all know I was going to say that??) The downside is that you have to 
>>> iterate through all your data once. The upside is a good one liner query 
>>> statement may be all you need to process your data. And if you need to make 
>>> multiple passes at your data, all the better. 
>>> 
>>> Bob S
>>> 
 On May 16, 2022, at 10:46 , Rick Harrison via use-livecode 
  wrote:
 
 I have a large chunk of data that I want to
 search as quickly as possible.  
 
 Unfortunately the part I want to search is the 
 middle third of the data.  The other thirds at 
 the beginning and at the end are just junk and 
 slow down my search so I want to get rid of them.
 
 I don’t want to search line by line as that
 takes way too long.
 
 There’s no unique character dividing any
 of these data regions.
 
 What’s the best way to do this?
 
 Thanks in advance!
 
 Rick
 
 
 
 ___
 use-livecode mailing list
 use-livecode@lists.runrev.com
 Please visit this url to subscribe, unsubscribe and manage your 
 subscription preferences:
 http://lists.runrev.com/mailman/listinfo/use-livecode
>>> 
>>> ___
>>> use-livecode mailing list
>>> use-livecode@lists.runrev.com
>>> Please visit this url to subscribe, unsubscribe and manage your 
>>> subscription preferences:
>>> http://lists.runrev.com/mailman/listinfo/use-livecode
>> 
>> ___
>> use-livecode mailing list
>> use-livecode@lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your subscription 
>> preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
> 
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Divide Large Data Blob?

[Bob Sneidar via use-livecode]

So this has got me thinking. Apparently what I am calling Divide and Conquer is 
really called a binary sort. I have looked up on the interwebs to calculate the 
maximum number of iterations for a given number of values, but it seems that 
all the formulas offered up use functions for C. I am trying to figure out what 
a basic math formula for this is, given n values. 

Bob S


> On May 16, 2022, at 15:23 , Bob Sneidar via use-livecode 
>  wrote:
> 
> A maximum of 7 recursions are necessary to isolate a single instance of 100 
> possible values. 1000 requires a maximum of 10. 1 values requires 14. The 
> idea is that for every factor of 10, you need roughly 3 more recursions. This 
> of course assumes the data is sorted, which in your case is sorted into 3 
> containers. If you know the limits of how many lines can be garbage, and how 
> many can be valid data, you narrow your scope significantly. 
> 
> Livecode is pretty damn quick at parsing this kind of data. If there are 
> consistent delimiters (in this case a line break) then even 20 or 30 
> recursions is child's play. 
> 
> Bob S
> 
> 
>> On May 16, 2022, at 15:00 , Bob Sneidar via use-livecode 
>>  wrote:
>> 
>> Do you know exactly which lines you need to toss, or do you need to searc 
>> the data to find out where the beginning and end of the useful data is? 
>> If the former, then just put line x to y of your data into a new variable. 
>> If the latter, then a divide and conquer approach might be the answer. Get 
>> the line 30% in, test for valid, get the line 40% in, test, then 35% then 
>> 32.5% or 37.5% depending on your test. 
>> 
>> You may only have to do this a dozen or so times to find the exact line 
>> where your valid data begins. 
>> 
>> The other way of course is to get it all into a SQL database (how did you 
>> all know I was going to say that??) The downside is that you have to iterate 
>> through all your data once. The upside is a good one liner query statement 
>> may be all you need to process your data. And if you need to make multiple 
>> passes at your data, all the better. 
>> 
>> Bob S
>> 
>>> On May 16, 2022, at 10:46 , Rick Harrison via use-livecode 
>>>  wrote:
>>> 
>>> I have a large chunk of data that I want to
>>> search as quickly as possible.  
>>> 
>>> Unfortunately the part I want to search is the 
>>> middle third of the data.  The other thirds at 
>>> the beginning and at the end are just junk and 
>>> slow down my search so I want to get rid of them.
>>> 
>>> I don’t want to search line by line as that
>>> takes way too long.
>>> 
>>> There’s no unique character dividing any
>>> of these data regions.
>>> 
>>> What’s the best way to do this?
>>> 
>>> Thanks in advance!
>>> 
>>> Rick
>>> 
>>> 
>>> 
>>> ___
>>> use-livecode mailing list
>>> use-livecode@lists.runrev.com
>>> Please visit this url to subscribe, unsubscribe and manage your 
>>> subscription preferences:
>>> http://lists.runrev.com/mailman/listinfo/use-livecode
>> 
>> ___
>> use-livecode mailing list
>> use-livecode@lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your subscription 
>> preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
> 
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Divide Large Data Blob?

[Bob Sneidar via use-livecode]

A maximum of 7 recursions are necessary to isolate a single instance of 100 
possible values. 1000 requires a maximum of 10. 1 values requires 14. The 
idea is that for every factor of 10, you need roughly 3 more recursions. This 
of course assumes the data is sorted, which in your case is sorted into 3 
containers. If you know the limits of how many lines can be garbage, and how 
many can be valid data, you narrow your scope significantly. 

Livecode is pretty damn quick at parsing this kind of data. If there are 
consistent delimiters (in this case a line break) then even 20 or 30 recursions 
is child's play. 

Bob S


> On May 16, 2022, at 15:00 , Bob Sneidar via use-livecode 
>  wrote:
> 
> Do you know exactly which lines you need to toss, or do you need to searc the 
> data to find out where the beginning and end of the useful data is? 
> If the former, then just put line x to y of your data into a new variable. If 
> the latter, then a divide and conquer approach might be the answer. Get the 
> line 30% in, test for valid, get the line 40% in, test, then 35% then 32.5% 
> or 37.5% depending on your test. 
> 
> You may only have to do this a dozen or so times to find the exact line where 
> your valid data begins. 
> 
> The other way of course is to get it all into a SQL database (how did you all 
> know I was going to say that??) The downside is that you have to iterate 
> through all your data once. The upside is a good one liner query statement 
> may be all you need to process your data. And if you need to make multiple 
> passes at your data, all the better. 
> 
> Bob S
> 
>> On May 16, 2022, at 10:46 , Rick Harrison via use-livecode 
>>  wrote:
>> 
>> I have a large chunk of data that I want to
>> search as quickly as possible.  
>> 
>> Unfortunately the part I want to search is the 
>> middle third of the data.  The other thirds at 
>> the beginning and at the end are just junk and 
>> slow down my search so I want to get rid of them.
>> 
>> I don’t want to search line by line as that
>> takes way too long.
>> 
>> There’s no unique character dividing any
>> of these data regions.
>> 
>> What’s the best way to do this?
>> 
>> Thanks in advance!
>> 
>> Rick
>> 
>> 
>> 
>> ___
>> use-livecode mailing list
>> use-livecode@lists.runrev.com
>> Please visit this url to subscribe, unsubscribe and manage your subscription 
>> preferences:
>> http://lists.runrev.com/mailman/listinfo/use-livecode
> 
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Divide Large Data Blob?

[Bob Sneidar via use-livecode]

Do you know exactly which lines you need to toss, or do you need to searc the 
data to find out where the beginning and end of the useful data is? 
If the former, then just put line x to y of your data into a new variable. If 
the latter, then a divide and conquer approach might be the answer. Get the 
line 30% in, test for valid, get the line 40% in, test, then 35% then 32.5% or 
37.5% depending on your test. 

You may only have to do this a dozen or so times to find the exact line where 
your valid data begins. 

The other way of course is to get it all into a SQL database (how did you all 
know I was going to say that??) The downside is that you have to iterate 
through all your data once. The upside is a good one liner query statement may 
be all you need to process your data. And if you need to make multiple passes 
at your data, all the better. 

Bob S

> On May 16, 2022, at 10:46 , Rick Harrison via use-livecode 
>  wrote:
> 
> I have a large chunk of data that I want to
> search as quickly as possible.  
> 
> Unfortunately the part I want to search is the 
> middle third of the data.  The other thirds at 
> the beginning and at the end are just junk and 
> slow down my search so I want to get rid of them.
> 
> I don’t want to search line by line as that
> takes way too long.
> 
> There’s no unique character dividing any
> of these data regions.
> 
> What’s the best way to do this?
> 
> Thanks in advance!
> 
> Rick
> 
> 
> 
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Divide Large Data Blob?

[Rick Harrison via use-livecode]

Hi Ralph,

Filtering the data first made a huge difference in the size of the data I need 
to search.

Thanks!

Rick

> On May 16, 2022, at 4:01 PM, Ralph DiMola via use-livecode 
>  wrote:
> 
> If the needed OR unneeded lines have something in common then the filter 
> command is your friend. Filter is blazingly fast.
> 
> Ralph DiMola
> IT Director
> Evergreen Information Services
> rdim...@evergreeninfo.net
> 
> -Original Message-
> From: use-livecode [mailto:use-livecode-boun...@lists.runrev.com] On Behalf 
> Of Rick Harrison via use-livecode
> Sent: Monday, May 16, 2022 1:46 PM
> To: How to use LiveCode
> Cc: Rick Harrison
> Subject: Divide Large Data Blob?
> 
> I have a large chunk of data that I want to search as quickly as possible.  
> 
> Unfortunately the part I want to search is the middle third of the data.  The 
> other thirds at the beginning and at the end are just junk and slow down my 
> search so I want to get rid of them.
> 
> I don’t want to search line by line as that takes way too long.
> 
> There’s no unique character dividing any of these data regions.
> 
> What’s the best way to do this?
> 
> Thanks in advance!
> 
> Rick
> 
> 
> 
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
> 
> 
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Divide Large Data Blob?

[Rick Harrison via use-livecode]

Hi Ralph,

I was looking at filter.  It might help me to
cut down on the amount of data to search,
but I would still have to throw out the
first 1/3 of the data.  Perhaps filtering
first and then processing what’s left
by line number wouldn’t be too slow.

Thanks for the suggestion!

Rick

> On May 16, 2022, at 4:01 PM, Ralph DiMola via use-livecode 
>  wrote:
> 
> If the needed OR unneeded lines have something in common then the filter 
> command is your friend. Filter is blazingly fast.
> 
> Ralph DiMola
> IT Director
> Evergreen Information Services
> rdim...@evergreeninfo.net
> 
> -Original Message-
> From: use-livecode [mailto:use-livecode-boun...@lists.runrev.com] On Behalf 
> Of Rick Harrison via use-livecode
> Sent: Monday, May 16, 2022 1:46 PM
> To: How to use LiveCode
> Cc: Rick Harrison
> Subject: Divide Large Data Blob?
> 
> I have a large chunk of data that I want to search as quickly as possible.  
> 
> Unfortunately the part I want to search is the middle third of the data.  The 
> other thirds at the beginning and at the end are just junk and slow down my 
> search so I want to get rid of them.
> 
> I don’t want to search line by line as that takes way too long.
> 
> There’s no unique character dividing any of these data regions.
> 
> What’s the best way to do this?
> 
> Thanks in advance!
> 
> Rick
> 
> 
> 
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode
> 
> 
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

RE: Divide Large Data Blob?

[Ralph DiMola via use-livecode]

If the needed OR unneeded lines have something in common then the filter 
command is your friend. Filter is blazingly fast.

Ralph DiMola
IT Director
Evergreen Information Services
rdim...@evergreeninfo.net

-Original Message-
From: use-livecode [mailto:use-livecode-boun...@lists.runrev.com] On Behalf Of 
Rick Harrison via use-livecode
Sent: Monday, May 16, 2022 1:46 PM
To: How to use LiveCode
Cc: Rick Harrison
Subject: Divide Large Data Blob?

I have a large chunk of data that I want to search as quickly as possible.  

Unfortunately the part I want to search is the middle third of the data.  The 
other thirds at the beginning and at the end are just junk and slow down my 
search so I want to get rid of them.

I don’t want to search line by line as that takes way too long.

There’s no unique character dividing any of these data regions.

What’s the best way to do this?

Thanks in advance!

Rick



___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Divide Large Data Blob?

[Rick Harrison via use-livecode]

Hi Richmond,

Doesn’t the following still require a loop?

LC doesn’t like the syntax you provided.

Thanks,

Rick

> On May 16, 2022, at 2:23 PM, Richmond via use-livecode 
>  wrote:
> 
> 3 put chars 1-SLICER of MyDATA into firstTHIRD

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Divide Large Data Blob?

[Rick Harrison via use-livecode]

Hi Richard,

I was looking at the offset function and thinking about
the starting points.  It still presents a looping problem
for me that I’m trying to avoid.  If other methods aren’t
more efficient I will play with it more.

Thanks,

Rick

> On May 16, 2022, at 2:32 PM, Richard Gaskin via use-livecode 
>  wrote:
> 
> The offset function has an optional third param for starting point, so 
> setting that value to the number of bytes you want to skip would take you 
> right where you want to be. Offset can then be used in a loop, updating the 
> starting value as you go. Exiting the loop once your into the tail you don't 
> want to index can be a simple if to escape the loop.
> 
> There may also be options for using delimiters. What does the source data 
> look like, and what should the resulting index look like?
> 
> And the most central question with things like this: how large is "large"?

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Divide Large Data Blob?

[Rick Harrison via use-livecode]

Hi Richmond,

An interesting approach.  I’ll give it try

Thanks!

Rick

> On May 16, 2022, at 2:23 PM, Richmond via use-livecode 
>  wrote:
> 
> Well one of the things you could do is this:
> 
> slightly pseudo
> 
> 1 put the number of chars in MyDATA into MyNUM
> 2 put MyNUM / 3 into SLICER
> 3 put chars 1-SLICER of MyDATA into firstTHIRD
> 4 put chars SLICER-(SLICER * 2) into secondTHIRD
> 5 put chars (SLICER * 2)-(SLICER*3) into thirdTHIRD
> 
> at which point you can perform you search inwith secondTHIRD.
> 
> Unless you want to recombine all those bits having performed some sort of 
> operation on secondThird
> lines 3 and 5 are a waste of time.
> 
> Best, Richmond.

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Decrypting (and encrypting) Large files

[Phil Davis via use-livecode]

On 5/10/22 12:35 PM, Richard Gaskin via use-livecode wrote:

Mark Clark wrote:

> Wondering if anyone has used LiveCode for encrypting-decrypting large
> files?
...
> I’m thinking about using LC for decrypting zip compressed log files
> that can be multiple gigabytes in size. I’d like to use just LC vs.
> resorting to shell if possible.

What is behind the preference to roll your own rather than call 
existing purpose-built command-line tools from LC with shell()?



I like this option. You could use shell("openssl ") to 
encrypt any size file. Also it may be possible to run 'openssl' as a 
process, by which (I believe) you could make it non blocking, running in 
the background. (I haven't tried opening 'openssl' as a process, just 
speculating here.)


Phil Davis




Your chunking described in your latest post seems the way to go, AFAIK 
pretty much how other tools would handle it.


Another option:  if you're in an environment where even log files 
require strong encryption, could you pipe log data to a separate 
secured log server instead?




--
Phil Davis
(503) 307-4363


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Divide Large Data Blob?

[Richard Gaskin via use-livecode]

Rick Harrison wrote:


I have a large chunk of data that I want to
search as quickly as possible.  

Unfortunately the part I want to search is the 
middle third of the data.  The other thirds at 
the beginning and at the end are just junk and 
slow down my search so I want to get rid of them.


I don’t want to search line by line as that
takes way too long.

There’s no unique character dividing any
of these data regions.

What’s the best way to do this?


The offset function has an optional third param for starting point, so 
setting that value to the number of bytes you want to skip would take 
you right where you want to be. Offset can then be used in a loop, 
updating the starting value as you go. Exiting the loop once your into 
the tail you don't want to index can be a simple if to escape the loop.


There may also be options for using delimiters. What does the source 
data look like, and what should the resulting index look like?


And the most central question with things like this: how large is "large"?

--
 Richard Gaskin
 Fourth World Systems
 Software Design and Development for the Desktop, Mobile, and the Web
 
 ambassa...@fourthworld.comhttp://www.FourthWorld.com

___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Divide Large Data Blob?

[Richmond via use-livecode]

Great Behinds Stink Alike.  :)

On 16.05.22 21:19, Craig Newman via use-livecode wrote:

Hi.

Can you get the number of lines of the whole blob, if lines are pertinent, 
divide that number by 3, and search from there? Another words, if you had 1000 
lines, divide by 3 and search from line 333 to 666.

Craig


On May 16, 2022, at 1:46 PM, Rick Harrison via use-livecode 
 wrote:

I have a large chunk of data that I want to
search as quickly as possible.

Unfortunately the part I want to search is the
middle third of the data.  The other thirds at
the beginning and at the end are just junk and
slow down my search so I want to get rid of them.

I don’t want to search line by line as that
takes way too long.

There’s no unique character dividing any
of these data regions.

What’s the best way to do this?

Thanks in advance!

Rick



___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode



___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Divide Large Data Blob?

[Richmond via use-livecode]

Well one of the things you could do is this:

slightly pseudo

1 put the number of chars in MyDATA into MyNUM
2 put MyNUM / 3 into SLICER
3 put chars 1-SLICER of MyDATA into firstTHIRD
4 put chars SLICER-(SLICER * 2) into secondTHIRD
5 put chars (SLICER * 2)-(SLICER*3) into thirdTHIRD

at which point you can perform you search inwith secondTHIRD.

Unless you want to recombine all those bits having performed some sort 
of operation on secondThird

lines 3 and 5 are a waste of time.

Best, Richmond.

Best Richmond.

On 16.05.22 20:46, Rick Harrison via use-livecode wrote:

I have a large chunk of data that I want to
search as quickly as possible.

Unfortunately the part I want to search is the
middle third of the data.  The other thirds at
the beginning and at the end are just junk and
slow down my search so I want to get rid of them.

I don’t want to search line by line as that
takes way too long.

There’s no unique character dividing any
of these data regions.

What’s the best way to do this?

Thanks in advance!

Rick



___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode



___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Divide Large Data Blob?

[Craig Newman via use-livecode]

Hi.

Can you get the number of lines of the whole blob, if lines are pertinent, 
divide that number by 3, and search from there? Another words, if you had 1000 
lines, divide by 3 and search from line 333 to 666.

Craig

> On May 16, 2022, at 1:46 PM, Rick Harrison via use-livecode 
>  wrote:
> 
> I have a large chunk of data that I want to
> search as quickly as possible.  
> 
> Unfortunately the part I want to search is the 
> middle third of the data.  The other thirds at 
> the beginning and at the end are just junk and 
> slow down my search so I want to get rid of them.
> 
> I don’t want to search line by line as that
> takes way too long.
> 
> There’s no unique character dividing any
> of these data regions.
> 
> What’s the best way to do this?
> 
> Thanks in advance!
> 
> Rick
> 
> 
> 
> ___
> use-livecode mailing list
> use-livecode@lists.runrev.com
> Please visit this url to subscribe, unsubscribe and manage your subscription 
> preferences:
> http://lists.runrev.com/mailman/listinfo/use-livecode


___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Re: Audio Control on Mobile?

[Jim Lambert via use-livecode]

> Dan wrote:
> My client wants to make a music playing app.  Need to replicate a music 
> player... so, we need to keep our [custom] transport in sync with the sound.  
> Stop, Start, Pause, Scrub, update current time and time remaining, etc.  EASY 
> on desktop, can't seem to do this on mobile with LiveCode.

I wonder if using the Browser widget would help?

https://developer.mozilla.org/en-US/docs/Web/Guide/Audio_and_video_delivery/Cross-browser_audio_basics
 


Jim Lambert
___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode

Divide Large Data Blob?

[Rick Harrison via use-livecode]

I have a large chunk of data that I want to
search as quickly as possible.  

Unfortunately the part I want to search is the 
middle third of the data.  The other thirds at 
the beginning and at the end are just junk and 
slow down my search so I want to get rid of them.

I don’t want to search line by line as that
takes way too long.

There’s no unique character dividing any
of these data regions.

What’s the best way to do this?

Thanks in advance!

Rick



___
use-livecode mailing list
use-livecode@lists.runrev.com
Please visit this url to subscribe, unsubscribe and manage your subscription 
preferences:
http://lists.runrev.com/mailman/listinfo/use-livecode