> If the form is externally linked and writeable in it's own DB, wouldn't
> it seem that that would be the logical place to write to?
Only if the architecture allows for multiple versions of that same
form to exist, to coincide with multiple versions of the original content
which contains it.
> This seems like a job for the conduit rather than the parser. One of
> our goals here is to eliminate the need for HotSyncing.
The "conduit" *IS* the parser on non-Windows platforms. We have tobe
careful not to architect out a large section of our userbase. Or, we develop
a conduit for the Unix folks again. Alexander and I hacked away at one quite
a while back, which still sits in the original archive, and did a
"real-time" sync of the records to the Palm from a local cache directory (or
from a stream in the later version). If we rely on conduit action, we need
to resurrect that.
> Its probably not the most useful for the rest of the Plucker community,
> but I'll share: an IE toolbar button launches a program which calls the
> spider then serially sends the file the parser made to the palm via a
> rudimentary protocol.
Again, using IE (granted, a large percent of the population) will
limit our target userbase to "lesser" functionality. Can this be done in
Java? Shockwave? Something more portable?
> The conduit and parser in our case become integrated, at least as it
> appears to the user. As these features more complex we're practically
> re-writing the HotSync manager, which seems like a lot of pointless
> work. But you do have complete control over exactly what happens, how
> easy it is, and how long it takes.
The new Hotsync Manager is a nightmare. Palm has decided to "shroud"
more of the previously-exposed functions in an API you must use to talk to
them, using their DLLs. Perhaps ideal in a Windows environment, but not if
you use some other tools (on Windows) other than VC++ or Java to do conduit
development (such as Cygwin or gcc for example).
> The form download should obviously happen while this transfer takes
> place. In the HotSync world, it'd be a conduit that does the work.
^^^^^^^ ^^^^^
Windows world
> It's the "allow the user to interact with it" part that still needs some
> more thought. How and when exactly are these form results going to be
> displayed? I propose some sort of "form manager" where you can get a
> list of each of the POSTs/GETs made to a form (including date, time, and
> data submitted) and have a chance to view any of the returned results to
> any of these submissions.
What about changing values already submitted before you sync?
Example: I sync, pull down a form to my palm. Fill it out, press
Submit, use my Palm for a week, decide I want to go change my "order" on
that form (such as an online shopping card order form or something, or
change my billing address), and fill out the form again (without ever having
sync'd a second time).
Do the original values get overwritten by my new values? Do they go
in a new FormsDB for each time I fill it out? Does the form show up as
"greyed out" when I return to it a second time, since I've already filled it
out?
Perhaps an Alert here?
[Form already processed, overwrite with new form data?]
> I think the architecture we're looking for is similar to the way email
> is handled on the Palm. When you send an email from Plucker, that email
> is dumped into the MailDB which then has a conduit on the desktop to
> deal with it. Forms should have their own DB, if not several, as you
> suggested, for each page they live off of.
For each *INSTANCE* of that page, including multiple versions for
each time I filled out the form (as above). Perhaps allowing multiple
versions of the Form to be filled out, then a way to view all versions, and
delete the ones you don't want, before sync. Have to think about this one a
bit more.
> > b. Where do we store the form data at sync time while a parser run is
> > gathering OTHER (non-posted form content)?
>
> What?
When the sync is started, and the form data is pulled from the Palm,
POST'ed, and then the response returned, where do we store that return
response, while the OTHER content is being sync'd? Let's say I fill out a
form on my Palm in Foo.pdb, and the data is stored in Foo-FormsDB-239.pdb,
and I also am creating a new pdb for Wired.com, which is being gathered at
the same time as the form is being processed. Where do we store that data,
and is the Palm in a 'sync' state while content is being GATHERED, or just
when it's being SYNC'd? (battery life, latency, etc.) Mike and I have tested
and build several-dozen megabyte databases, which take quite a long time to
gather and build, I don't want my Palm sitting in a cradle for 5 hours while
these things are running. Do we require the user to sync twice; once to
query and extract any form data to POST, and then again to actually push the
new pdb(s) onto the Palm?
> > c. How do we support SSL-based forms in the current Python scheme,
> > without requireing the user to recompile Python with libssl?
>
> Oh dear god. Second version, please... :)
A would wager that a large percentage of forms that are being filled
out are behind an SSL wrapper. Online ordering, stock quote queries, bank
statements, all of that is behind SSL (and if it's not.. I would hope nobody
is using them). It's going to be something asked for, once people realize we
have forms support.
> Cookies would be nice, except they have usually have expiration times so
> trying to use them at next sync would fail. At this point, we're going
> to have to have *some* level of cooperation from the web site itself.
> Either non-expiring cookies or a unique ID embedded in the form which
> would get returned with the form results.
We'll never get support from the websites, because we'll never know
who is sync'ing what to what website. MAYBE some can detect the
HTTP_USER_AGENT string, and use some other cookie based on that, but it
would be a very small percentage. Cookies are used for session. We may be
able to encode the cookie in the URI, but that too would require some more
things Python doesn't have natively (and fortunately for me, perl does).
We could also modify the cookie, not ideal, but it could be done.
$expiry = $expiry + 3 months;
> > e. How do we handle the precedence of forms in a depth-first vs.
> > breadth-first gathering process?
>
> I'm not sure what you mean here. Forms live on individual pages; what
> sort of order is there between them?
Observe the example in my previous email to Chris about the way my
bank handles the online banking logins for a good example.
> Like I said, it seems more appropriate for the conduit to be managing
> the records from the Palm. The conduit should be able to feed the
> returned HTML directly into the parser. Hrmmmmm.... yeah, I'm still not
> comfortable with this. It seems to me that the parser should be just
> that, a parser, and not have to worry about pulling stuff from the palm.
We should separate this functionalty:
1. Conduit: talks to Palm, identifies content, pulls/pushes data
2. Parser: groks nothing other than HTML and web-based content,
feeds that to the conduit.
> The parser currently could read in HTML that has been dumped to a local
> file, but then it'd have a local, incorrect URL attached to it, yes?
No, write the data to a cache file, like we used to in the earlier
versions of Plucker, where we used to create a cache file on disk of each
graphic and html page, then concatenate them into one pdb at sync time with
conduit.pl.
> It'd be nice to be able to feed the parser some HTML code and assign a
> specified URL to it. I really want to be able to select a block of text
> in a browser and be able to transfer only that portion to the Palm,
> which we could do if this was implemented.
That's definately something you have to handle on the Windows side,
interacting with the clipboard, something like:
HGLOBAL XferItemsToClipboard(CStringArray& ClipboardList, CPoint&
MousePosition)
{
HGLOBAL GlobalMem = XferItemsToGlobalHeap(ClipboardList, MousePosition);
if (GlobalMem) {
if (::OpenClipboard(NULL)) {
VERIFY(::EmptyClipboard());
VERIFY(::SetClipboardData(CF_HDROP, GlobalMem));
VERIFY(::CloseClipboard());
}
else {
::GlobalFree(GlobalMem);
}
}
return GlobalMem;
}
..but I aren't a Windows programmer.
/d
