Re: DXR with clang not showing callers/callees
Does it give you an error message? For me, running make in that directory now only builds the test outputs, it doesn't actually run the tests. For that you need to run make test from the top-level directory. Btw, `make` in the test dir never ran tests AFAIK: https://github.com/mozilla/dxr/blob/26a7353903d3f98d5d30531a019ad4be1226e1c1/tests/json-test/makefile. Unfortunately the more wideranging code reorganisation seems to have broken things, and I've been unable to get anything working since then. Try a git pull, and then make clean make test at the top level, and let me know what goes wrong. I don't want you to have to spend time chasing this. I'm in #static on irc.mozilla.org as ErikRose all day if you need me. Thanks, Erik ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: DXR with clang not showing callers/callees
Ah, you know what I forgot? I should say Go do `python setup.py develop` (if you want to hack on it) or `python setup.py install` before doing anything. somewhere. On the Vagrant VM, which is where I do my work, I've rigged that to happen automatically. Sorry! Btw, installing the package as above should pull down its dependencies automatically, and there should be no need to do anything to the PYTHONPATH. Cheers, Erik ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: DXR with clang not showing callers/callees
I should say Go do `python setup.py develop` (if you want to hack on it) or `python setup.py install` before doing anything. somewhere. Done. Feel free to submit pulls to improve clarity. https://github.com/mozilla/dxr/commit/e0a3b50d7ea0cfa55c73dcb0a7eb5112bedb5e40 ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: is there a verbose or log everything option to dxr-build?
I'm working off the testing branch, pulled yesterday, and when I try to build it complains and tells me that there is no depend target. That's a lie, of course, because I use the depend target often in my make system, so obviously dxr-build is not finding the makefile properly. If this does turn out to be a dxr bug, let's add a regression test when we fix it. I want to start battening down the test situation so we can change things without breaking them. The problem is that I think I've told it everything correctly in the config file (and this same config worked with a pull from the same branch a couple of weeks back). One of the two of us is wrong. I can't find any kind of verbose mode, though. The only thing I get in the build.log is that make couldn't find a depend target. There's no option, but it's pretty noisy already. You can add more print statements or breakpoints (import pdb;pdb.set_trace()) to build.py to get more. Erik ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: clang plugin and clang/llvm 3.2?
That's what's strange...git status doesn't complain about the plugins directory in the old location. In fact, I can do a git log in that directory and get stuff from as recently as Jan 15. My mistake. My guess is that git, which tracks only files, doesn't bother with that plugins dir if it contains only that .so, because *.so is in .gitignore. Bah. ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: clang plugin and clang/llvm 3.2?
I guess the take-away lesson is that when doing a re-org of the tree, there needs to be some out-of-band way to communicate to anyone who may have a clone to perform a clean. I'll be sure to post about that and stick it in the (coming soon) changelog in the future. Sorry to had to chase this. Erik ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: DXR Shorthand search
Thanks Erik No problem! Ah, and the + in front of some of those queries means use the fully qualified name, and require an exact match. (Thanks, abbeyj.) Erik ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: DXR Shorthand search
I should also mention that we're thinking about not having contains searches happen by default: https://github.com/mozilla/dxr/pull/104 ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: DXR Shorthand search
I have a Firefox keyword for searching with old DXR. Is there a hope we can enter search criteria directly in the url? I don't see why not. Hit Return in the search field, and you'll get a URL like this: http://33.33.33.77:8000/search?tree=codeq=readmeredirect=true You could easily sub the query in for the q= query param. The only questionable bit would be url escaping, which FF handles well. However, I could see the + sign getting interpreted as a space. Erik ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: Error on first make DXR
I know you got this fixed, but here's my guess at what happened for the benefit of future victims. Puppet provisioning probably timed out while downloading LLVM and clang. You would have seen a subtle error report about this when doing vagrant up. Unfortunately, Puppet doesn't stop on error, as it's meant for production scenarios where The Show Must Go On. :-) Erik On Mar 4, 2013, at 6:55 AM, Schalk Neethling sneethl...@mozilla.com wrote: Got my vagrant machine up and running but when running make inside ~/dxr on the VM, I get the following: Welcome to your Vagrant-built virtual machine. Last login: Fri Sep 14 06:22:31 2012 from 10.0.2.2 vagrant@precise32:~$ cd ~/dxr/ vagrant@precise32:~/dxr$ make make -C dxr/plugins/clang build make[1]: llvm-config: Command not found make[1]: Entering directory `/home/vagrant/dxr/dxr/plugins/clang' make[1]: llvm-config: Command not found g++ -Wall -Wno-strict-aliasing -c dxr-index.cpp -o dxr-index.o dxr-index.cpp:1:27: fatal error: clang/AST/AST.h: No such file or directory compilation terminated. make[1]: *** [dxr-index.o] Error 1 make[1]: Leaving directory `/home/vagrant/dxr/dxr/plugins/clang' make: *** [build-plugin-clang] Error 2 ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: DXR Shorthand search
The text search in MXR does a contain search by default, and the identifier search does an exact match. I would try to match that. There could be a syntax for exact match, like foo Good news: abbeyj fixed this last week, though it isn't deployed yet: https://github.com/mozilla/dxr/pull/104. (Really looking forward to that continuous deployment environment slated for Q2.) BTW : I remember queries were very fast once, but they seem to be slow again, cf http://dxr.allizom.org/search?q=type-ref%3Adroptree=mozilla-central Yikes, that is slow. Filed: https://bugzilla.mozilla.org/show_bug.cgi?id=847582 And to start to display results only once the search is completely finished. Should be pretty easy. Filed: https://bugzilla.mozilla.org/show_bug.cgi?id=847580 Great feedback! Thanks! ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: help for the python challenged: python setup.py develop failing
I am not an idiot, but assume that for anything related to python I am ignorant but read well. I love this sentence and might use it in the future. :-) So, I get to the point of doing python setup.py develop, and it chugs along, then stops trying to download Flask. Specifically, I get the following error: Searching for Flask=0.9 Reading http://pypi.python.org/simple/Flask/ Reading http://github.com/mitsuhiko/flask/ Best match: Flask 0.9 Downloading http://pypi.python.org/packages/source/F/Flask/Flask-0.9.tar.gz#md5=4a89ef2b3ab0f151f781182bd0cc8933 error: MD5 validation failed for Flask-0.9.tar.gz; possible download problem? I presume that the md5 is an expected hash value for the downloaded file, but I can't figure out where it's getting that value or why it might be failing. Huh, I don't remember ever seeing that before. Presumably it pulls it off PyPI, where it's linked from the https://pypi.python.org/pypi/Flask/ page—search for md5. I just tried python setup.py develop in a new virtualenv and wasn't able to reproduce your problem. If you're still having it, try installing everything with pip first, manually, before doing setup.py develop: pip install -r requirements.txt python setup.py develop Maybe that will work better/differently. Let me know if you keep having trouble. I'm on IRC. Cheers, Erik ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: help for the python challenged: python setup.py develop failing
Doing the pip first worked, though I had to lengthen the timeout delay. This machine sits behind a firewall that actually downloads everything on your behalf first, then scans it for viruses/issues, then returns it to you if it passes all checks. Ooo, fancy-pants. :-) Rebuild worked just fine. Good! virtualenv is kind of cool; may have to learn a bit more about that. Yep, I don't do anything without that, lest I muck up my global environment and then wonder why everything on my box acts weirdly from then on. Is the vagrant setup that included able to set up a complete environment in a virtual machine such that I wouldn't have to configure my personal desktop (or some other server) to be the DXR server? I didn't really intend the VM to act as a production server, but I don't see any hard-and-fast reason that it couldn't. The Apache config is fine, and if it doesn't suit your needs, it's at least a good example. Two other issues: 1. callers:someFunc never completes for me; I get a 504 Gateway Timed Out Is there an easy way to run the same dxr request from the command line to see if it's a timeout problem or a query issue? actually, is there a CLI-level interface at all? Are you accessing DXR through Apache, then? I think if you run the dev server with dxr-serve.py and talk straight to it, it won't time out. It sounds like we're missing an index or something. 2. when dxr-build finishes generating the files/directory hierarchy into 'target_folder', it doesn't put any of the .htaccess files in there. I have to copy them over from my previous target directory. Should they be generated into there, or should I expect to have to replenish them anew each time? We don't bother with htaccess files anymore. Static files are no longer served directly by Apache, as I found that it didn't make enough of a speed difference to matter. See the comments in httpd/conf.d/dxr.conf: # We used to make special efforts to also serve the static pages of # HTML-formatted source code from the tree via plain Apache, but that # tangle of RewriteRules saved us only about 20ms per request. You can do # it if you're on a woefully underpowered machine, but I'm not maintaining # it. 3. I also find that the .htaccess files that are part of the distribution are insufficient with regard to Options required. I need both ExecCGI and FollowSymLinks in order for it to work. That might be a consequence of my apache config, but even so it might be worthwhile to add that Options line into the boilerplate dot-htaccess and just comment it out by default. I assume you're talking about the old htaccess files you pulled from the old build. There shouldn't be any anymore. You can take all those fancy RewriteRules out of your Apache config. ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: help for the python challenged: python setup.py develop failing
To be clear, the only thing I need to do is run dxr-serve.py, e.g., $ dxr-serve.py ?? dxr-serve.py target (where target is the folder spat out by dxr-build) All the commands have good help now: you can just pass them --help. ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: help for the python challenged: python setup.py develop failing
Tried this and it worked. callers:someFunc took, according to the returned web page Query executed in 225.399s That's a long time. How can I help you figure out which index (or indices) are missing? If I remember correctly, that query used to have an obnoxious number of outer and inner joins. Maybe there's a way to denormalize the data a bit such that the query could be simplified? Try sticking explain=True in the query string, and see if that reveals anything. ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: Can dxr still pull from version control?
In a really old and ancient version of dxr, one could configure it to pull in changes from version control as part of the build. Is that still in there somewhere and I'm not seeing it? This might have been part of DXR itself in the past, though I have no knowledge of it. Currently, we have a shell script that checks out the mozilla-central sources in an ad hoc manner and then kicks off the DXR build: http://hg.mozilla.org/build/tools/file/b2fae78a2db9/scripts/dxr/dxr.sh. There are some concrete benefits to intertwingling DXR more with version control. For one thing, it would be one way to do incremental builds (https://github.com/mozilla/dxr/pull/121), which would drastically drop our build times. Our hg plugin already interacts with hg a little bit to do blame links and such. If you have a proposal in which VCS integration would be a big benefit, I'm happy to listen. Of course, we'd want to stay decoupled enough to be able to support any arbitrary VCS, so there are certainly benefits to keeping our scope small as well. Erik ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: dxr.mozilla.org seems unusable
David, I am fixing it as we speak! Due to some IT bottlenecks, it's been limping along on an obsolete version for months now, missing all the good bug fixes and improvements that have landed on master. We've tried to apply makeshift fixes, but those have made things worse in some cases. I'm grinding off the cracked and pocked road surface and doing a deeper paving job now. I'll post details once it's done—today, if all goes well. Expect dxr.allizom.org to stabilize first, followed by dxr.mozilla.org. Thanks for your patience! Erik On May 28, 2013, at 10:36 AM, David Rajchenbach-Teller dtel...@mozilla.com wrote: Dear list, Not quite sure it's the right list for this, but for at least one week, I have been unable to use dxr.mozilla.org – links don't work. With whom should I get in contact? Thanks, David -- David Rajchenbach-Teller, PhD Performance Team, Mozilla ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: dxr.mozilla.org seems unusable
David et al, I've completely redone the staging site, http://dxr.allizom.org/ . It should be useable; please try it out. My next 2 orders of business are to reproduce this work for the production site, dxr.mozilla.org, and to bring the moz-central tree up to date—currently, it's serving a March 11 build due to difficulties getting newer ones to build under clang. dxr.allizom.org is now serving the very tip of master, which means all the fixes and improvements of the last 4 months are finally available to enjoy. What's more, I have sketches of a continuous-deployment script that will keep it that way. And all is serving via a proper pool of WSGI workers rather than old-school CGI. We're still croaking along on a RHEL 5.5 box, but I managed to pull together enough modern libraries that everything seems happy. Please try to break it! Thanks again for your feedback and patience as I wrestle this deployment under control. Next, dxr.mozilla.org, a recent moz-central, and continuous deployment! Erik ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
dxr.mozilla.org is updated!
DXR's production site (dxr.mozilla.org) is now running the latest version of DXR, on a modern bed of pooled WSGI processes. This follows yesterday's update of the staging site and brings with it 4 months of feature additions and bug fixes. Both sites are still using a March 11 version of the mozilla-central tree, so figuring out clang weirdnesses and getting that up to date are my next orders of business. Then, continuous deployment, features, UX, and fun! Best regards, Erik ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: dxr.mozilla.org seems unusable
A word of warning about the hg plugin, it includes the log message once for each line... This might blow up the HTML file size more than it already is... So you might want to checkout file size of a big cpp files before you enable this in production. Thanks! I think Taras mentioned that to me a few weeks ago. I'll be doing local builds pretty regularly for some time, so I'll be sure to notice anything of the sort. (I think there is a suggestion in plugins/hg/htmlifier.py about refactoring annotations, etc.) But it might be better to lazy load blame information on demand. And yes, now that we're not driving everything off a single CGI and taping things together with RewriteRules, things like this get much easier to manage. :-) Erik ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Yesterday's mozilla-central now serving on DXR
Both sites are still using a March 11 version of the mozilla-central tree No more! dxr.mozilla.org is now serving a mozilla-central from only 18 hours ago. I finally got everything building under clang. There is yak hair all over the place. Next come automatic updates for both the DXR code and moz-central. In the meantime, I'll try to build moz-central regularly on my local VM and upload it manually. Cheers, Erik ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: dxr.mozilla.org is updated!
Fixed on master, and my experimental continuous deployment script rolled it out to dxr.allizom.org. I'm going to do a few more commits to work out any kinks and then expand it to production. On Jun 2, 2013, at 2:16 , Jonas Finnemann Jensen jop...@gmail.com wrote: Looks to me as if the newly added /source/ part of the urls are missing a few places... For example: dxr/static/search.js#L324 Maybe it's not the only place... ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
New DXR wireframes
I just finished a set of wireframes rethinking DXR's UI. (This is actually revision 2; many of you probably didn't see revision 1.) Please take a look and send feedback! https://wiki.mozilla.org/DXR_UI_Refresh Specifically, I'm looking for things that will bother you in your common workflows. Also keep an eye out for complexity that could be removed without making typical operations difficult. I've tried to add explorability and learnability for new users while maintaining or improving speed of use for the familiar crowd. And there are several new considerations that make nav and the multi-tree story much better for everybody: * Case-insensitive searching * Visiting the same line of a file in a different tree * Searching within a browsed-to path * Making it easy to tell what file you're looking at Best regards, Erik ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: Poll: What do you need in MXR/DXR?
Thanks, everyone, for your thoughtful and voluminous input! I bequeath you the following behind-the-scenes links so you can see the effect your feedback is having on DXR's future. We've collected, de-duped, and categorized your feedback at https://wiki.mozilla.org/DXR_UI_Refresh#Feedback. If you see your request grossly mischaracterized or omitted, feel free to edit. I'm watching the page. I've done a first cut at immediate goals just above that, at https://wiki.mozilla.org/DXR_UI_Refresh#Plans_And_Priorities. Finally, there's a rough draft at a revised (and more sensical) query syntax up at https://github.com/erikrose/dxr/blob/query-parser/docs/queries.rst. This is just the beginning. I'll be posting updates at https://blog.mozilla.org/webdev/tag/dxr/ whenever there's something useful to say, and you can watch the DXR_UI_Refresh wiki page if you want more frequent updates. Cheers, Erik ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: Poll: What do you need in MXR/DXR?
It was fun hanging out at the summit... I like the simplifications, but have 3 quick comments: 1) You forgot the exclusion prefix -, for queries like: -excludedkeyword -excluded phrase -path:*.cpp Good catch. Fixed! 2) I saw fully qualified names under To Be Determined I hope you keep them around for links. Oh, yes, I plan to. If you read carefully, you'll see that only the spelling is to be determined. :-) Due to whitespace issues and macros they're not useful for the user... Oh really? I was not aware of that and would benefit from elaboration. 3) Consider adding multiline search to To Be Determined :) If that's not on the wiki, please add it. We'll probably need to do a bunch of index work before adding that to the query syntax will do any good. :-) Erik ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
DXR UI refresh is live!
https://blog.mozilla.org/webdev/2014/02/07/dxr-gets-a-huge-ui-refresh/ Happy hacking! ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: DXR UI refresh is live!
that's pretty, but where did the list of classes class members go when viewing files like http://dxr.mozilla.org/mozilla-central/source/modules/libjar/nsZipArchive.cpp Coming back soon: https://bugzilla.mozilla.org/show_bug.cgi?id=965659. We're having some trouble reducing it to a test case smaller than moz-central. ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: DXR UI refresh is live!
Hi, Reed. Rest assured, many of your wishes are planned for the future. But when you have a big project, you've got to pick some things. Last week's release was an important milestone: it was the minimal feature set that would yield a cohesive experience under a UI new design. The new design solves a lot of usability problems, removes some serious limitations, and gives us places to put several things planned for the near future. Now, ordinarily, I abhor big-red-lever launches, as evidenced by our continued rollout of features onto production while the UI work was in progress and the fact that we deployed 7 times yesterday. However, you can't always iterate your way to big user-facing changes and make sense the whole time. So don't take the current state as an end; it's just an uncharacteristically large step, worthy of a release announcement. Let me address your individual wishes: * Where are the non-mozilla-central repositories? Coming! One of the big UI-branch features is improved support for multiple trees. The UI for that is hidden right now, since there's only one tree indexed. But we're working on comm-central right now. Aurora is next. Then we'll continue pulling in more, in order of the popularity they see on MXR. I think you'll like what you see. * Where's the integration with CVS? * Where are the links to see changes over the last day, week, month, etc.? A lower priority since there are plenty of other ways to poll version control. However, we have a flexible framework for VCS integration (https://github.com/mozilla/dxr/blob/master/dxr/plugins/omniglot/htmlifier.py), and this would be an easy first bug, if it's important to you. (If you can get a bunch of people to hoot and holler about how important it is, we'll move it up the roadmap.) * Why do I have to click on Navigation just to get basic functionality? That was already ticketed, and I fixed it yesterday; the nav pane is now disclosed by default. More is planned: https://bugzilla.mozilla.org/show_bug.cgi?id=968020. * Why is it not accessible over SSL? There's an IT bug filed about that, but indexing more trees is higher priority right now. Do you have a specific use case in mind? Feel free to have a look at our roadmap (https://wiki.mozilla.org/DXR_Roadmap) and add constructively to it. Cheers, Erik ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: DXR UI refresh is live!
that's pretty, but where did the list of classes class members go when viewing files like http://dxr.mozilla.org/mozilla-central/source/modules/libjar/nsZipArchive.cpp I believe this is fixed now, a casualty of a needless use of JS, which is now gone. If you still see this behavior, please reopen https://bugzilla.mozilla.org/show_bug.cgi?id=965659. Thanks for riding out the bumps! Erik ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
DXR now indexing comm-central
We've just added comm-central to DXR! You can find it at http://dxr.mozilla.org/comm-central/source/ or through the new Switch Tree menu. The menu knows some tricks: it tries hard to preserve things like your browsing position and your search query while moving from tree to tree, so you can chase similar lines of inquiry across branches. This will become even more thorough in the future: for instance, pinning it so you can flit between branches while keeping your eyes locked at the same line of a file. This marks the beginning of our conquest of All The Trees. More are coming: https://bugzilla.mozilla.org/show_bug.cgi?id=820531. Thanks to Kendall Libby for getting the build going and Joshua Cranmer for sage advice on the build system. For now, it's indexing as if it were building SeaMonkey, but we'll be switching to Thunderbird as soon as we get that working. There are only about 16 files' difference between the two, so most of you probably won't notice. With wishes of happy hacking, Erik ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Multi-line highlighting landed
Thanks to contributor Jamon Camisso, multi-line highlighting has just landed in DXR. Now you can do things like this... http://dxr.mozilla.org/mozilla-central/source/CLOBBER#10-15,18-22,4-5 ...by pressing buttons like this: http://dxr.readthedocs.org/en/latest/code-highlighter.html Enjoy! Erik ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: clang version required for DXR
Thanks, everyone! One further question, is there anything preventing us from working with the current clang ToT? Have there been severe API changes since we wrote this code? The best way to know is to try. :-) Clang's C++ API changes from minor version to minor version. We've had to add ifdefs every time since 3.2, so my guess is that there have been changes. They don't even try to keep it stable, shunting you to the plain C API (which is less capable) if you want stability. I'd certainly take patches toward top-of-tree support; we're going to need them eventually. Cheers, Erik ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: Issue while setting up DXR with vagrant.
From the Killed, it looks like you're running out of memory and the OS has started killing off processes to keep things afloat. The set of measurements at the bottom of your transcript shows only 33% memory use, but I wonder if I could get a second opinion: what does vmstat -S m -a say on the VM before and after a failed make? I cannot find any piece of code in the VM Take a look in /home/vagrant/dxr. Is there code in there? I think you will find that there is, as that's where peep.py lives, mounted with the rest of the code as part of a Virtualbox shared folder. How much RAM does the host machine have? We have one other contrib with problems running DXR on a very small host. You could try increasing RAM on the VM. I don't see why it should work in CI for us with the stock settings but fail for you, but it's worth a try. Thanks for taking the time to ask for help. We'll get it figured out. :-) Cheers, Erik ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
DXR Intern Presentation
Marcell has been my intern on the DXR project these past months, and it's been a privilege to work with him. From JavaScript analysis to Elasticsearch to writing Haskell in Python, he's cut an impressive swath through the codebase, and you can expect to see a lot of his handiwork deployed in the next several weeks. He'll be giving a bit of a preview in a wide-ranging presentation this Thursday. Oh, and there might be dinosaurs. And robots. 8/21 @1:30PM PST in Mountain View (will be streamed on AirMo) Topics include... - Type signature search - JavaScript support almost done - Moving to Elasticsearch - DXR fan art! We hope to see you there! (There will also be an accompanying blog post going up after the talk on mvcisback.github.io.) Cheers, Erik DXR Lead ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: DXR on FreeBSD
(per IRC) so that did help, and I uploaded the CSV to a github gist: https://gist.github.com/rhelmer/c3a2a2defcdb98ed2152 So is this just an insane type? Absolutely; that is utterly insane. Wow. That's more like a compiler stress-test than an actual type. I am not sure if it's a bug... do you want a patch to bump up the CSV field_size_limit? Maybe a compromise would be to make it configurable... I was really hoping it was going to be a CSV-emitting bug on our side: that we missed a quotation mark or something. But since it's not, sure, I'll take a field-size-limit-increasing patch. I don't think there's really a need to make it configurable. The crazy values you're running into are 200K. If we made the limit 500K, I don't think anybody would notice the RAM hit. And, if you can do a quick test to make sure the _csv module isn't allocating a static buffer of the max size, larger would be fine as well (though we should stop short of sys.maxsize, which we were kicking around earlier; that's around 7 exabytes ;-)). Cheers, Erik ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: MDN Compat Service via static analysis
We've been working on a browser compatibility data store API [2][3]. To that, we (think we) need to add a linting tool that will use the compat data store to make a code compatibility report. The question for this group is - can we get someone who knows static analysis involved now to help us make sure the compat data model is conducive to generating rules for the linter? I'm happy to consult. Want to set up 20 minutes to chat sometime? I think it'll be a lot more efficient to do a bursty, high-bandwidth session than for me to try to reconstruct your brain contents from etherpads. For CSS and HTML, you'll barely have to do anything deserving of the term analysis; I imagine you'll mostly be checking attribute and property names. But there will be some parsing, and that's one of my favorite things. JS will, of course, be hard and should probably wait for 2.0. We're working on that as well, as part of DXR, so we can share some code. For our 20-minute powwow, how about you bring compatibility schema ideas, I'll bring parsing and analysis knowledge, and we'll figure out how to make them meet in the middle? Cheers, Erik ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: Web Compat Service Static Analysis
Everything David says is on the level. For analyzing code out in the wild,
esprima might be a good candidate to build on. For analyzing Firefox's internal
JS (which is what we're trying to do in DXR), it's not suitable, since
Spidermonkey went off in all sorts of non-standardized directions, and no
off-the-shelf tool supports ES6 yet, let alone our pre-standardization
dalliances.
Marcell, who interned on DXR this summer, might have something to add here, as
he did a great deal of shopping for JS analysis tools before coming up empty.
He even had a plugin running for awhile based on ternjs, which does some neat
object-shape-tracing tricks. If we're thinking about combining our efforts, I
see two possibilities: go with something like esprima and give up ES6 and
Spidermonkey compatibility for the near future, or add analysis capabilities to
Spiderflunky (https://github.com/erikrose/spiderflunky), which is based on
Spidermonkey's own JS parser (with the advantage that we don't have to write it
in JS).
Can anyone who works on Firefox say whether we're headed toward ES6 conformance
internally and, if so, what the timeline might be?
Cheers,
Erik
On Nov 7, 2014, at 14:55 , Luke Crouch lcro...@mozilla.com wrote:
Thanks David, that's very helpful.
I'm adding dev-static-analysis to this to see if anyone on that list
(Erik maybe?) has any ideas on this.
In short, we're learning that JS will certainly be the most challenging
language for which to create a compatibility linter/scanner. ;)
-L
On 11/7/14 10:18 AM, David Bruant wrote:
Le 07/11/2014 16:42, Luke Crouch a écrit :
+Justin
Thanks for the info David, sorry you coudn't make it.
In addition to the updated notes in the etherpad [1], Justin and I met
[2] with Thijs Busser who created http://jscc.info/ that mashes up data
from Kangax Can I Use with a bunch of regexs.
He mentioned wanting to use esprima.org? Do you like that approach too?
Totally. Anything that relies on regexps to do anything with code is
doomed to fail at some point. AST-based analysis are the way to go
(fundamentally, code as a data structure is a trees and regexps cannot
work on trees).
And esprima is the state-of-the-art JS parser that lots of static
analysis tools are based on (escodegen, eslint, etc. The list of
dependent tools on npm is saying https://www.npmjs.org/package/esprima
). Esprima is fully ES5-compatible and they're actively working on an
ES6 branch (harmony), so that would be the parser I'd bet on.
But in my opinion, it still won't be enough. A naive tool based on
esprima won't be able to understand that the following is not
browser-compatible (yet, because Object.assign is introduced in ES6):
var O = Object;
var assign = O.assign;
assign({}, {});
(and some forms of this will occur in real-life code, because on the
web, everything happens)
To understand that this code is not ES5 compatible, one has to track
that the O variable has such type (and such static functions attached to
it), etc. and that's tedious work. The good news is that it's exactly
the sort of work the TypeScript compiler was designed to do.
Other compiler like the Closure Compiler do this kind of work too, but
they force you into writing your JavaScript in a particular style (in
order to make the code more easliy analysable). The reason I'm so fan of
TypeScript is that they've found a great balance of analysing types in
JS while letting authors write their code in whatever style fits them.
Hope that helps,
David
Thanks,
-L
[1] https://etherpad.mozilla.org/compat-data-for-static-analysis
[2] https://devengage.etherpad.mozilla.org/jscc-mdn
On 11/7/14 8:58 AM, David Bruant wrote:
Hi Luke,
I'm very sorry, I couldn't attend the meeting. I was catching up after
MozFest.
As far as JavaScript static analysis is concerned, I'm sure there is a
quick win to be won hacking the TypeScript compiler (especially now
that they've re-written it entirely and that it's apparently much
cleaner).
Tools like JSLint/JSHint/ESLint (3rd is my favorite because of its very
modular rule architecture) can verify some aspects of a program, but are
really syntax-based. For instance, these tools cannot tell whether
you're using setTimeout with a string or a function as first argument.
On the other hand, TypeScript can and does and you can be sure it does,
because that's what it's been written for (track types across variables
and function arguments).
TypeScript is initialized with a declaration file so that the compiler
knows about the standard library (functions exposed per ECMA and W3C
specs). I'm sure it would be easy to change these declaration files to
see if some code is using non-standard and/or edgy (or prefixed)
functions in their code.
rhaa... I'm babbling and it's complicated to explain in an email.
If you're interested, maybe we can call some time next week so I explain
all that (unless it's
Re: vagrant up help
Hi, Andrew. I haven't seen this one before. Do you have a reasonably powerful machine? Have you tried vagrant destroy followed by another vagrant up, in case something went awry during initialization? If neither of those helps, you could try adding the suggested config directive to Vagrantfile: put config.vm.boot_timeout = 600 after config.vm.box = ubuntu/trusty64, and see if more time does the trick. Btw, you very probably want to be on the es branch if you're not already; that's DXR 2.0, where all the development is these days. It'll shortly be merging into master. Cheers, Erik DXR Lead On Jun 9, 2015, at 5:48 , Andrew Artajos andrew.artajos@gmail.com wrote: Hi dev-static-analysis, I'm following DXR docs and I got stuck in the part where $ vagrant up it displays an error: http://mibpaste.com/ni0VCs I hope you could help. -- ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: DXR 2.0 staged. Feedback please!
It looks like finding of overrides of virtual methods is missing from DXR 2.0. Is this intentional? Hmm, no. The tests seem to pass (https://github.com/mozilla/dxr/blob/es/dxr/plugins/clang/tests/test_overrides.py). Where are you seeing it? ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
DXR 2.0 And Beyond: A Whistler Session
Dear friends with large codebases— You are heartily invited to a DXR 2.0 pony show and design session next Friday in Whistler, where we'll demonstrate the new abilities of this near-rewrite, answer questions, and figure out what goes next on the roadmap. For those not familiar with it, DXR is a code search and browsing tool that indexes not only the text of your program but its structure as well, using data exfiltrated from compiler or parsers. It's also one of the few engines out there that will let you run arbitrary regexes across a large codebase in tenths of a second. We'll demo… * Improved C/C++ analysis * Multi-language support—Python and Rust, for starters * Generic identifier search * Image browsing * Zero slow queries * Independent, parallel tree indexing. (This will help us get all the trees currently under MXR indexed.) * Permalinks to old revisions * A new plugin architecture so we can add new languages, query types, and cross references easily (https://dxr.readthedocs.org/en/es/development.html#writing-plugins) Now that the infrastructure is solid, we can strike out in many interesting directions. Come and campaign for what you want! Just a few ideas: * Cross-project search * Arbitrary boolean queries * Search by color * JS support * IDL support * Useful visualizations Add it to your Whistler schedule: http://sched.co/3gWR Cheers, Erik Rose DXR Project Lead ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
Re: Listing all function declarations
> Is there a way to search using filters AND wild cards in DXR? For example, > if I wanted to list all the methods defined on There isn't right now, but it's something we could add. Can you describe the sorts of tasks you'd use it for? Cheers, Erik ___ dev-static-analysis mailing list dev-static-analysis@lists.mozilla.org https://lists.mozilla.org/listinfo/dev-static-analysis
