Re: Encountering slow navigation to a card containing very large fields? Do this one simple trick
Let me know off list if you want to have a look at the handlers I have written for getting data from a datagrid array into a memory database and back again so your queries can be more efficient. Bob S > On Apr 21, 2022, at 04:50 , David V Glasgow via use-livecode > wrote: > > > Thanks to all who responded on this. I will definitely take a look at > Datagrids as a possible solution. Both Phil and Jim suggested avoiding > navigation completely …. which I will also investigate. > > Cheers > > David G > >> On 21 Apr 2022, at 2:23 am, Jim Lambert via use-livecode >> wrote: >> >>> DavidG wrote: >>> That is exactly it, Craig! Text processing and field responsivity on the >>> card containing the fields are as fast as I could hope. >>> >>> It is simply navigating back to the card containing the fields that can >>> take several seconds, depending on how much text is in the fields. The >>> workaround is to have invisible groups containing the field(s) placed on >>> the card you are navigating from. >> >> Here's another possibility. >> Instead of using multiple cards, do the whole interface on a single card. >> Have different groups on that single card, each of which does the job of one >> of the eliminated cards. >> In that way you NEVER NAVIGATE away from your data fields. They are simple >> present in a now hidden group. >> >> For example, let's say your stack currently has 5 cards. >> Card #1 is the card with the two data fields you describe. >> Put those fields and any other interface elements currently on that card >> into Group #1 of the card. >> Put the contents of cards 2 thru 5 into groups #2 thru 5 on card #1. >> Hide and show groups where you previously navigated between cards. >> >> Jim Lambert >> >> ___ >> use-livecode mailing list >> use-livecode@lists.runrev.com >> Please visit this url to subscribe, unsubscribe and manage your subscription >> preferences: >> http://lists.runrev.com/mailman/listinfo/use-livecode > > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your subscription > preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Encountering slow navigation to a card containing very large fields? Do this one simple trick.
Maybe context would help. I receive digital evidence from police Hi Tech Crime Units, consisting of lists/tables of messages, search histories, web histories, files downloaded, etc etc. These relate to online activities of alleged internet offenders. Some consist of data relating to a few months activity, others may be 10 years or more - so vary hugely in the sheer quantity of data. This is what is imported into the first field. I have established a number of sets of keywords/tags which can be cycled through using ‘filter with’ to find list items which might indicate particular forms of malign motivation of the alleged offender. So, a user clicks button “Filter with X keywords” looking for, for example, coercive/aggressive tags. All ‘filtered with' lines containing selected tags appear in the second scrolling field. Sometimes nothing much will be found, but sometimes this field may become 50 to 74% the size of the field containing the original imported data. This (second) field can be scrolled and eyeballed until something of forensic significance is spotted. Clicking on that line scrolls the first field to display and highlight the line in its original context (i.e showing preceding and following lines). With respect to Bob’s observation about the work to display a background group being 'already done’ if the navigated-to card contains the background group containing the field, that is pretty much what I assumed is happening. The intriguing question is why the work necessarily is ‘undone’ when navigating to any card not containing the group. Does it consume such significant resources the default is to clear it? It would be nice to be able choose to preserve or cache the work if that has such a significant impact on performance as in this case. Best Wishes, David Glasgow > On 18 Apr 2022, at 7:02 pm, Richard Gaskin via use-livecode > wrote: > > David Glasgow wrote: > > > The user just initiates the automatic filtering for various keywords. > > What does that mean? What actions do they perform requiring them to review > the full list? > > -- > Richard Gaskin > Fourth World Systems > Software Design and Development for the Desktop, Mobile, and the Web > > ambassa...@fourthworld.comhttp://www.FourthWorld.com > > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your subscription > preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Encountering slow navigation to a card containing very large fields? Do this one simple trick.
David V Glasgow wrote: > On 16 Apr 2022, at 8:20 pm, Richard Gaskin wrote: >> >> How many is "many many thousands of lines"? 10k? 100k? More? > > Typically 30,000 to 800,000. What is the user asked to do with 800,000 lines of data? That is, does this need to be displayed? >> Is the data displayed in list fields, or must the line wraps be >> calculated? > > The wrapped line field is locked and doesn’t change after import > and is unaffected by the keyword-containing-lines process. I > suppose a good question is are line wraps recalculated on navigation > to the card. I'd guess that the card record is unpacked before rendering, which would include the controls, such as fields, and their contents. -- Richard Gaskin Fourth World Systems ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Encountering slow navigation to a card containing very large fields? Do this one simple trick.
> On 17 Apr 2022, at 12:30 am, Bob Sneidar via use-livecode > wrote: > > Yes, I think you have it when you say that displaying text in a field is > processor intensive. Fields are notoriously inefficient at storage. Do all > your processing in memory and then display your results in whatever object > you desire. Yup, I do that. processing takes time, but honestly it is amazingly fast. Dumping found lines into the results field takes a surprising amount of time, but even that doesn’t bother me too much. The issue is purely navigating back to the card containing the 600lb fields from any card that doesn’t. > Datagrids can provide an efficient intermediary display mechanism because the > data is already in a form that is easy to work with, namely arrays. An array > is merely a handle to a position in memory. A datagrid is a window on that > object. As you scroll through the datagrid, you are only manipulating the > data appearing in the window, and not the data itself. > > A field is different in that how the data appears in the interface must be > calculated in advance. Think of it like the old “How do you eat an Elephant” > allegory. The Datagrid eats the Elephant, “One Bite At a Time.” The field > tries to eat the Elephant all at once. > > Sent from my iPhone > >> On Apr 16, 2022, at 01:40, David V Glasgow via use-livecode >> wrote: >> >> Thanks for this, Bob. I have followed various discussions about datagrids, >> but didn’t appreciate the property your “window on the data” metaphor >> describes. I have also noticed that processing text in a variable is >> amazingly fast but then the results being put into a field takes a lot of >> time. I assume the linking of a text variable to a datagrid would also >> avoid that bottleneck? >> >> That is very interesting and I will explore the functions you describe. >> Thanks very much. >> >> Best Wishes, >> >> David Glasgow >> >> >>> On 15 Apr 2022, at 8:35 pm, Bob Sneidar via use-livecode >>> wrote: >>> >>> Not sure why you are having this issue, but I will say that this is one of >>> the primary reasons Datagrids were created. Table fields with large amounts >>> of data were causing issues, and especially if the limits of lines or >>> characters were exceeded. >>> >>> With a Datagrid however, only the data that is visible is actually being >>> displayed. For example, you can have a datagrid with 10 visible rows and >>> containing 10,000 lines of text (or array keys if you prefer) but only 10 >>> lines of data will actually be in fields in the datagrid. There is wizardry >>> in play when you scroll. It *appears* like there is data above and below >>> the visible data, but trust me there is not. You can think of a datagrid as >>> a moving window above the data. Anything not showing throught the window is >>> not being processed by your vision, if you take my clumsy analogy. >>> >>> Once the data is in the datagrid, there are a number of ways to work with >>> it. I like using the dgData of the datagrid which returns a numbered array, >>> because I have written functions like filterArray and arrayToSQLite, >>> cursorToArray, arrayToText, mergeArray etc. which allow me to work >>> efficiently with these numbered arrays. >>> >>> Of particular interest to you might be the arrayToSQLite, bcause once you >>> have data in an sqLite table, querying is a much more powerful (and >>> efficient) way of getting just the data you want, such as the information >>> displayed in your summary page. >>> >>> To use datagrids, you only need to know about a few properties, >>> particularly the dgData, the dgText, the dgHilitedLine / Index, the >>> dgDataOfLine / Index and handling the selectionChanged message. >>> >>> Bob S >>> >>> > On Apr 15, 2022, at 10:27 , David Glasgow via use-livecode > wrote: Hi folks, MacOS 10.13.6 and Livecode 9.6.0 It may be that this is well known to everyone but me. But just in case... I have a card (A) with two fields in separate groups. Field 1 contains many many thousands of lines of text which are filtered for keywords and the results displayed in field 2 (which can also be many thousands of lines). The user can navigate to a separate card (B) to view summary statistics and a bar chart. I found that worked fine, but navigating back from B to A always took many seconds and looked like a hang. The delay seemed to be in some way proportional to the number of lines in field 1 and field 2. By a slip of the finger I discovered that this lag disappears completely if the groups containing field 1 and 2 are also placed on the summary stats card B. There is no real need for either to be there otherwise , and it seems they can even be invisible. I tried to see if there is a significant memory overhead to doing this, but there doesn’t seem to
Re: Encountering slow navigation to a card containing very large fields? Do this one simple trick.
Hi Richard > On 16 Apr 2022, at 8:20 pm, Richard Gaskin via use-livecode > wrote: > > How many is "many many thousands of lines"? 10k? 100k? More? Typically 30,000 to 800,000. One field wrapped (the imported ‘source’ text) and the other a, unwrapped list of found lines containing a keyword. > > Is the data displayed in list fields, or must the line wraps be calculated? The wrapped line field is locked and doesn’t change after import and is unaffected by the keyword-containing-lines process. I suppose a good question is are line wraps recalculated on navigation to the card. > > -- > Richard Gaskin > Fourth World Systems > Software Design and Development for the Desktop, Mobile, and the Web ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Encountering slow navigation to a card containing very large fields? Do this one simple trick.
Hi Jim, The text in fields doesn’t move or change after the initial analysis. the slowdown is just navigating back to to the card containing the fields from from one which doesn’t contain the fields. Cheers David G > On 16 Apr 2022, at 8:46 pm, Jim Lambert via use-livecode > wrote: > > >> but then the results being put into a field takes a lot of time. > > Does locking the screen before putting text into the field help speed things > up? > ___ > use-livecode mailing list > use-livecode@lists.runrev.com > Please visit this url to subscribe, unsubscribe and manage your subscription > preferences: > http://lists.runrev.com/mailman/listinfo/use-livecode ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Encountering slow navigation to a card containing very large fields? Do this one simple trick.
So fields are snakes and datagrids are mice. -- Jacqueline Landman Gay | jac...@hyperactivesw.com HyperActive Software | http://www.hyperactivesw.com On April 16, 2022 6:32:19 PM Bob Sneidar via use-livecode wrote: The Datagrid eats the Elephant, “One Bite At a Time.” The field tries to eat the Elephant all at once. ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
Re: Encountering slow navigation to a card containing very large fields? Do this one simple trick.
David Glasgow wrote: > I have a card (A) with two fields in separate groups. Field 1 > contains many many thousands of lines of text which are filtered > for keywords and the results displayed in field 2 (which can also > be many thousands of lines). The user can navigate to a separate > card (B) to view summary statistics and a bar chart. > > I found that worked fine, but navigating back from B to A always > took many seconds and looked like a hang. The delay seemed to be > in some way proportional to the number of lines in field 1 and > field 2. How many is "many many thousands of lines"? 10k? 100k? More? Is the data displayed in list fields, or must the line wraps be calculated? -- Richard Gaskin Fourth World Systems Software Design and Development for the Desktop, Mobile, and the Web ambassa...@fourthworld.comhttp://www.FourthWorld.com ___ use-livecode mailing list use-livecode@lists.runrev.com Please visit this url to subscribe, unsubscribe and manage your subscription preferences: http://lists.runrev.com/mailman/listinfo/use-livecode
