Re: leo4sqlite: table imports/exports ready for testing
+1 AsciiDoc! I still prefer reST, though. On Tue, Dec 12, 2017 at 1:56 PM, Matt Wilkiewrote: > > Feature wise I prefer restructured text, but unfortunately I feel it's >> lost the race in too many widespread tools. > > > Faint voice from back of theater "AsciiDoc!" > "TortoiseHG!" > "Fossil!" > > ok, I'll shut up now. > > -- > You received this message because you are subscribed to a topic in the > Google Groups "leo-editor" group. > To unsubscribe from this topic, visit https://groups.google.com/d/ > topic/leo-editor/Nqn5GHAIS1o/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > leo-editor+unsubscr...@googlegroups.com. > To post to this group, send email to leo-editor@googlegroups.com. > Visit this group at https://groups.google.com/group/leo-editor. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
Re: 90f9865: the NewShowData class
On Wednesday, December 13, 2017 at 6:03:07 PM UTC-6, Edward K. Ream wrote: > Here are the good parts. Actually we can make analyze much faster by not using any generator at all: def analyze(self, fn, root): ast_d = { ast.Assign: self.assigns_d, ast.AugAssign: self.assigns_d, ast.Call: self.calls_d, ast.ClassDef: self.classes_d, ast.FunctionDef: self.defs_d, ast.Return: self.returns_d, } fn = g.shortFileName(fn) for d in ast_d.values(): d[fn] = [] for node in ast.walk(root): d = ast_d.get(node.__class__) if d: d[fn].append(self.format(node)) Edward -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
Re: 90f9865: the NewShowData class
On Wednesday, December 13, 2017 at 6:11:15 PM UTC-6, Edward K. Ream wrote: > g.printDict({key: sorted(set(d.get(key) for key in d}) A last-minute edit went bad. This should be: g.printDict({key: sorted(set(d.get(key))) for key in d}) Edward -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
Re: 90f9865: the NewShowData class
On Wednesday, December 13, 2017 at 6:03:07 PM UTC-6, Edward K. Ream wrote: > The new class uses Vitalije's suggested patterns. show_results can be simplified with a dict comprehension: def show_results(self): ... for (name, d) in table: print('%s...' % name) g.printDict({key: sorted(set(d.get(key) for key in d}) So this is good :-) Edward -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
90f9865: the NewShowData class
The new class uses Vitalije's suggested patterns. This @button node tests the code: import imp import leo.core.leoCheck as leoCheck imp.reload(leoCheck) files = leoCheck.ProjectUtils().leo_core_files() leoCheck.NewShowData().run(files[:2]) Here are the good parts: def analyze(self, fn, root): table = ( (self.assigns_d, (ast.Assign, ast.AugAssign)), (self.calls_d, ast.Call), (self.classes_d, ast.ClassDef), (self.defs_d, ast.FunctionDef), (self.returns_d, ast.Return), ) self.fn = fn = g.shortFileName(fn) for d, ast_types in table: d[fn] = [] for node in ast.walk(root): for d, types in table: d[fn].extend(self.visit(node, types)) def format(self, node): class Formatter(leoAst.AstFormatter): level = 0 s = Formatter().visit(node) line1 = g.splitLines(s)[0].strip() return g.truncate(line1, 80) def show_results(self): '''Print a summary of the test results.''' table = ( ('assignments', self.assigns_d), ('calls', self.calls_d), ('classes', self.classes_d), ('defs', self.defs_d), ('returns', self.returns_d), ) for (name, d) in table: print('%s...' % name) d2 = {} for key in d: d2[key] = sorted(set(d.get(key))) g.printDict(d2) def visit(self, node, types): if isinstance(node, types): yield self.format(node) And that's it. Quite a contrast from the old ShowData and ShowDataTraverser classes. Edward -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
Re: What I've just learned from Vitalije
On Wednesday, December 13, 2017 at 3:28:50 PM UTC-6, Edward K. Ream wrote: There are several coding patterns in Vitalije's prototype that deserve > mention. > *Generators as predicates* > > Vitalije's prototype (as modified in leoCheck.py) contains this: > > for node in ast.walk(root): > classes_list.extend(do_class(node)) > Similar code works for dicts. This code: def gen(): for i in range(2): s = str(i) yield ('file'+s, 'val'+s) return None d = {} d.update(gen()) g.printDict(d) prints: { file0:'val0', file1:'val1' } This is likely to come in handy in rewriting the ShowData class. Edward -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
Re: ENB: What's next for check-conventions
On Wed, Dec 13, 2017 at 12:32 PM, Edward K. Reamwrote: I was going out the door when I wrote my first brief replies. Several questions deserve longer answers. I don't like too many tracing code to be left inside code. >> > > I understand, but for now it helps me understand what I *want* to do. > Furthermore, I have been thinking of this project as a way to peer into existing code. Let's use the term *reporter code* for this. I'll create the reporter code using the ast.walk + generator pattern or perhaps a rewritten ShowData class. Using separate reporter code should eliminate many traces. I still don't know how you plan to resolve names to types. >> > > These are the difficult resolve methods. > It's easy for me to forget that this is the *one and only* important question! In essence, the present code special-cases ivars injected from outside the class. They deserve special attention, but that special case is no substitute for a more general plan. As I think of this, it seems that the goals of the project are extremely limited, namely to check assignments (including correspondence of call args with signature) in *only* those case where naming conventions are used. As presently understood, inferences can only be as good or detailed as the data in the default classes dictionaries. Otoh, there are only a few official ivars (and chains of same) that Leo devs need to know about, so *maybe* these kinds of limited inferences *between* classes will suffice. One can imagine almost unlimited inferences *within* each class, but I really haven't put much thought into this. The disappointment I suffered a few days ago arises, I think, from the limited nature of the checks. > >> Python 3.6 has added support for type annotations. >> > I didn't make clear how much I dislike Python's type annotations. They may be useful in some cases, but imo they destroy the beauty of Python code. They have all the charm of C++ templates. They are likely to make refactoring code significantly harder. You could say that this project arises from the belief that Leo's naming conventions say almost as much annotations. Hope this clarifies my present thinking. I don't have a deep attachment to this project. It's already improved existing code, and the recent patterns you shared with us promise to collapse the complexity of one or more classes. Other than that, it may be that I'll abandon this project in just a few days. Edward -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
What I've just learned from Vitalije
There are several coding patterns in Vitalije's prototype that deserve mention. *Generators* ['abc'].extend(None) throws a TypeError. However, the following works: def a_generator(): if 0: yield 'abc' # Makes the function a generator return None aList = ['abc'] aList.extend(a_generator()) print(aList) # prints ['abc'] Returning None from a generator is, iirc, equivalent to raising StopIteration. *Generators as predicates* Vitalije's prototype (as modified in leoCheck.py) contains this: for node in ast.walk(root): classes_list.extend(do_class(node)) This is exquisite code. It's perfect for summarizing code. I shall rewrite the ShowData class using this pattern. It should be possible to eliminate the ShowDataTraverser class, a big collapse in complexity. Note that in this case the fact that ast.walk traverses the tree in "no particular order" does not matter. *Using defaultdict* This code is elegant. The first arg to defaultdict is a default factory function. All other args are passed to the dict constructor. from collections import defaultdict default_factory = lambda:dict(ivars={}, methods={}) known_classes = defaultdict(default_factory, **known_classes0()) Edward -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
Re: ENB: What's next for check-conventions
On Wed, Dec 13, 2017 at 2:03 PM, vitalijewrote: > You can start by studying the resolve methods. >> > > Ok, I am studying it and have a question. In resolve_ivar method if > ivar=='self' it returns ('class', class_name). Shouldn't it return > ('instance', class_name)? > Yes. Thank you. > > Little bit further in same method it checks if ivar is among special_names > ('c', 'p', 'g', 'v', ...) and if it is it returns spec_obj which is > ('instance', class...). It seems to me that return values for self and c, > p, g, v ... are not consistent. Either in both cases it should return > ('instance', ) or in both cases ('class', ...). > > Am I missing something? > No. You are right. The present code should use 'instance' everywhere. Edward -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
Re: ENB: What's next for check-conventions
I see you have added doc strings in c/self_assign functions that says they yield for each assignment node. Actually they yield single value if input node is assignment of correct type or don't yield at all. They expect to be called once for each node, they do not traverse tree. Perhaps it would be more readable not to yield data, but to return single element list or empty list. Vitalije -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
Re: ENB: What's next for check-conventions
> > You can start by studying the resolve methods. > Ok, I am studying it and have a question. In resolve_ivar method if ivar=='self' it returns ('class', class_name). Shouldn't it return ('instance', class_name)? Little bit further in same method it checks if ivar is among special_names ('c', 'p', 'g', 'v', ...) and if it is it returns spec_obj which is ('instance', class...). It seems to me that return values for self and c, p, g, v ... are not consistent. Either in both cases it should return ('instance', ) or in both cases ('class', ...). Am I missing something? Vitalije -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
Re: ENB: What's next for check-conventions
On Wed, Dec 13, 2017 at 11:50 AM, vitalijewrote: I didn't like the Type class. It can be replaced with named or even > ordinary tuple. > We'll see... > I couldn't find that you have instantiated it anywhere with last two > optional arguments. > Old cruft. > Also it defines method `__eq__`, but it is never used (or I didn't find > usage). > The code crashes without it. I don't like too many tracing code to be left inside code. > I understand, but for now it helps me understand what I *want* to do. I still don't know how you plan to resolve names to types. > These are the difficult resolve methods. > I don't see how it can be relevant for other cases? > That's what I am trying to understand. > Python 3.6 has added support for type annotations. > I am aware of annotations. I've written a tool that helps create annotations automatically. I've abandoned the idea of static type checking. However, the rope package does a pretty good job of that. It might be useful to add support for rope to Leo. If you have a concrete data structure that would allow type checking, and > that you can share (even if it is hand written for simple code example), I > would be glad to further develop my prototype for extracting all required > data from source code in a format that you need. > The present code makes some inferences. You can start by studying the resolve methods. Edward -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
Re: ENB: What's next for check-conventions
> > P.S. A nit. I dislike "ok and" pattern. For example, I would rewrite > is_assign_to this way: I don't mind your way of rewriting it. I used it just because I didn't know in advance all checks that need to be fulfilled, and `ok = ok and ...` gave me opportunity to put trace between any two and see what was the reason that some function returned False. Also, at some point I can use `ok = ok or `. I am glad that you liked this approach. If I knew more precisely what data you want to have after second pass, I would go for it. I didn't like the Type class. It can be replaced with named or even ordinary tuple. I couldn't find that you have instantiated it anywhere with last two optional arguments. Also it defines method `__eq__`, but it is never used (or I didn't find usage). It looks as if you thought it would be needed, but until it is really needed it just consumes brain power. I don't like too many tracing code to be left inside code. It consumes a lot brain cycles to read it and it hides real effective code. For example: all methods do_... take as an argument regex match and then again they throwaway input argument and compute same match once again. You said that patterns are defined where they are used, but it seems to me, that it was the case at some earlier stage, and now it is not the case. It is not as obvious mostly due to the presence of tracing code. If you turn on all of the traces I believe it would produce so much output that it would be challenging to read and understand traces. IMHO, traces should be always on while debugging one or two functions, but as soon as debugging is over, I would delete traces. For me it is much easier to delete and retype them even ten times if necessary than to read them all the time. Typing-in code is much cheaper than studying it. I still don't know how you plan to resolve names to types. I see relatively straightforward way to check and find the error in given example code, but I don't see how it can be relevant for other cases? Python 3.6 has added support for type annotations. Maybe that is the best way to define and check type conventions. When I ran my code on whole Leo code base I was puzzled by the fact that there is just a few places where pattern c. = self is used. Notes-->@file ../doc/leoAttic.txt-->Unused code-->Static type checking... As you can see under 'Unused code'. How useful would be leoCheck in the end? Hopefully, I am missing something. If you have a concrete data structure that would allow type checking, and that you can share (even if it is hand written for simple code example), I would be glad to further develop my prototype for extracting all required data from source code in a format that you need. Vitalije -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
Re: ENB: What's next for check-conventions
On Wed, Dec 13, 2017 at 7:44 AM, vitalijewrote: > Here is my attempt to show in code what I couldn't explain with the words. > It looks like you forgot these lines in my_check_func: for x in self_assign(_n): s_assigns.append(x) As you can see, I am studying the code closely. There are many elegant, useful ideas. Edward -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
Re: ENB: What's next for check-conventions
On Wednesday, December 13, 2017 at 7:44:57 AM UTC-6, vitalije wrote: Here is my attempt to show in code what I couldn't explain with the words. > ... > current version successfully adds entries to map of known_classes. In > particular it adds injected theTagController to the Command class ivars. > > Value that my code adds for each of those entries is AST node, containing > all relevant information. It can be examined further when resolving. > Many thanks for this excellent work. The code snippets are great tools. I'll use something like this as the basis of the production version. It may take awhile for me to adjust to this way of thinking, but it is obviously better than my previous ast code. Edward P.S. A nit. I dislike "ok and" pattern. For example, I would rewrite is_assign_to this way: def is_assign_to(node, _id): ''' returns True if this node is assignment to some attribute of variable named _id. Something like: `_id. = ...` ''' if isinstance(node, ast.Assign) and len(node.targets) == 1: arg = node.targets[0] return ( isinstance(arg, ast.Attribute) and isinstance(arg.value, ast.Name) and arg.value.id == _id ) else: return False And I also prefer docstrings to comments ;-) P.P.S. The new ProjectUtils. leo_core_files method is preferable to the leoFiles list. EKR -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
Re: ENB: What's next for check-conventions
Here is my attempt to show in code what I couldn't explain with the words. Attached to this message you can find Leo document with script node that analyzes little bit modified your test example which demonstrates wrong method call. My code in the current version successfully adds entries to map of known_classes. In particular it adds injected theTagController to the Command class ivars. Value that my code adds for each of those entries is AST node, containing all relevant information. It can be examined further when resolving. Vitalije -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout. LeoCheck-vitalije.leo Description: Binary data
Re: ENB: What's next for check-conventions
On Tuesday, December 12, 2017 at 1:06:50 PM UTC-6, Edward K. Ream wrote: As you can see from my lack of comments on other matters, this project requires my undivided attention. Recent work: 1. The "kind" switch in checker.check is one of those small things that makes a surprisingly big difference. It replaces several switches, and is mostly self documenting. 2. The prototype can report many useful data. You could call it another version of the ShowData class. 3. Early this morning I failed in the first attempt to generalize do_assn_to_c. I'll try again today. 4. It's still unclear how useful this tool will be. Two details relate to this question: - The code now suppresses the creation of entries for "special" classes, those classes for which the use has supplied type information. The idea is that the user has specified the public interfaces for those classes, so inferences should use no other data. This suppression also simplifies dumps of symbol tables. - Otoh, the code *must *handle *injected ivars*, that is, ivars that are set from outside the class. Handling injected ivars is essential for handling the original bug. Without this kind of knowledge the user would have to supply way too much information. Furthermore, reporting injected ivars is likely to be useful, if for no other reason that it would suggest other knowledge that the user could supply automatically. 5. Recent changes work around parsing errors in the regex's. There is no way that regex can do a full, accurate job of parsing python. I am considering creating a version of the prototype that uses the ast in a more traditional way. It will be interesting see how big an impact this switchover will have. Otoh, the switchover is considerably less important than inference-related work, so I may restrain myself a little longer. 6. There is a weird bug in the rewritten ProjectUtils.project_files method. When first reloading Leo it can fail to find any files in the 'leo' project. As I write this, I see that the present code is brain dead. For leo, it imports leo.core and uses leo.core.__file__ to find the base directory. But Doh, that's just g.app.loadDir! *Summary* - assn_to_c should be generalized. That will allow the prototype to report injected ivars. - The resolve code should be simplified. - It's nearing time to transition to ast-based code. Edward -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
Re: Installing Pyenchant
On Tue, Dec 12, 2017 at 3:52 PM, Matt Wilkiewrote: > > This probably merits an issue in the tracker. I'll go start one. >> > > https://github.com/leo-editor/leo-editor/issues/636 > Thanks for this. I've learned not to worry about the number of outstanding issues ;-) Edward -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.
Re: Summary of installation discussions
Hi Matt, I'm testing the pip3 install leo process. I prepared my system, simulating a new system with a pip uninstall for both PyQt5 and docutils from my python folder, then: > pip3 install leo It seems to be a developers version of Leo and certainly installed all the required packages. Here is the tail end of the install log: [snip] Installing collected packages: pyflakes, requests-toolbelt, pkginfo, tqdm, twine, docutils, pywin32-ctypes, keyring, babel, imagesize, snowballstemmer, sphinxcontrib-websupport, alabaster, sphinx, pypandoc, PyQt5, leo Successfully installed PyQt5-5.9.2 alabaster-0.7.10 babel-2.5.1 docutils-0.14 imagesize-0.7.1 keyring-10.5.0 leo-5.7.dev406 pkginfo-1.4.1 pyflakes-1.6.0 pypandoc-1.4 pywin32-ctypes-0.1.2 requests-toolbelt-0.8.0 snowballstemmer-1.2.1 sphinx-1.6.5 sphinxcontrib-websupport-1.0.1 tqdm-4.19.5 twine-1.9.1 c:\~\AppData\Local\Programs\Python\Python36> So I just typed leo at my ConsoleZ window and Leo appears! That’s great. I did begin to wonder if anything was working though, so maybe it would be nice for users to have a simple feedback message like ‘starting leo’ even if just a glimpse. Ah, now I understand what leoc is for. I'll add some notes about the log in console feature. "LaunchLeo.py is not needed at all after installing with pip." And it doesn't work; so I won't refer to it in the pip install instructions. Here is leo log: Leo Log Window Leo 5.6, build 20171213015234, Wed Dec 13 01:52:34 CST 2017 Git repo info: branch = master, commit = a5e1e740e704 Python 3.6.4, PyQt version 5.9.3 Does the build match what you would expect installing from the pypi release? Regards Lewis On Wednesday, December 13, 2017 at 8:24:55 AM UTC+11, Matt Wilkie wrote: > > > This is exactly what *leo *and *leoc* do. They are executable wrappers > around launchLeo.py* formatted in a way to make them work for whatever OS > it's installed on. The only requirement is that PYTHONHOME\Scripts be in > PATH. > > * well more specifically the code inside launchLeo. LaunchLeo.py is not > needed at all after installing with pip. > > matt > >> > -- You received this message because you are subscribed to the Google Groups "leo-editor" group. To unsubscribe from this group and stop receiving emails from it, send an email to leo-editor+unsubscr...@googlegroups.com. To post to this group, send email to leo-editor@googlegroups.com. Visit this group at https://groups.google.com/group/leo-editor. For more options, visit https://groups.google.com/d/optout.