Re: [Haskell-cafe] GSoC proposal: Haskell AST-based refactoring and API upgrading tool
On 29 Apr 2013, at 07:00, Niklas Hambüchen wrote: I would like to propose the development of source code refactoring tool that operates on Haskell source code ASTs and lets you formulate rewrite rules written in Haskell. Seen this? http://www.haskell.org/haskellwiki/HaRe Regards, Malcolm ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] GSoC proposal: Haskell AST-based refactoring and API upgrading tool
* Niklas Hambüchen m...@nh2.me [2013-04-29 14:00:23+0800] I would like to propose the development of source code refactoring tool that operates on Haskell source code ASTs and lets you formulate rewrite rules written in Haskell. Hi Niklas, This is a great idea. I talked about it at HIW last year[1] and have been working on it since then. [1]: http://www.youtube.com/watch?v=Ae-6uIMQPmU 1. What you call a full-source AST is already present in haskell-src-exts (under Annotated subtree). It /almost/ satisfies pretty . parse = id, (one thing it fails to do is to preserve tabs), and fixing it should be definitely easier than starting from scratch. 2. HSE has two pretty-printers: an exact one, and one that ignores annotations and pretty-prints in its own style. It's important to have a mixture of both, so that we can pretty-print generated snippets and splice them into an already formatted AST. A friend of mine, Pavel Poukh, is working on a pretty-printer that produces an annotated tree instead of a flat string, and AFAIK that work is close to done. [2]: https://github.com/Pnom/haskell-ast-pretty 3. A major thing that I devoted my time to is name resolution and package management [3,4], which are necessary for the tool to work. [3]: https://github.com/feuerbach/haskell-names [4]: https://github.com/feuerbach/haskell-packages They are also close to done. 4. As for the tool itself, I have a crude prototype [5], but I haven't updated it for a while. [5]: https://github.com/feuerbach/hasfix 5. The user interface is, of course, an important topic by itself. I was thinking of something less expressive and more declarative, but your idea is also interesting. Let's discuss this separately. Regarding this topic as a GSoC project, I'm not so sure this is a good idea. As a past GSoC student myself, I feel that new projects (as opposed to established projects with existing user base and development team) have much greater risks. (This is also backed up by data [6].) But if you (or anyone else who happens to read this) would like to get involved, please get in touch with me! [6]: http://www.gwern.net/Haskell Summer of Code Roman ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] GSoC proposal: Haskell AST-based refactoring and API upgrading tool
Hello Malcolm, no, I had indeed not seen this! Thanks for the link. It goes very much in the direction I was thinking of, but it does not seem to maintained and does not cabal install either. It also seems very much focused on interactive editor integration as compared to written-out transformations. Do you know to what extent they have built and a modification-friendly AST? Also, do you know if the people involved in this are still active in the community and interested in working further in this direction? Thanks Niklas On 29/04/13 15:36, Malcolm Wallace wrote: On 29 Apr 2013, at 07:00, Niklas Hambüchen wrote: I would like to propose the development of source code refactoring tool that operates on Haskell source code ASTs and lets you formulate rewrite rules written in Haskell. Seen this? http://www.haskell.org/haskellwiki/HaRe Regards, Malcolm ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] GSoC proposal: Haskell AST-based refactoring and API upgrading tool
There is another aspect to this: How do you get maintainers to apply the patches? How should hackage be changed to accomodate large-scale refactorings? There was a discussion on this mailing list related to build regressions on GHC 7.6 last year. All of the regressions could be fixed using perl regexps, and it was only a few hours of work, much less than the work involved in the discussion itself. I downloaded all of hackage and did the fixes using perl. http://www.haskell.org/pipermail/haskell-cafe/2012-August/103155.html However, without the community infrastructure to actually apply the patches, the problem is not solved. I think this is mainly a community/organizational issue. Refactoring is not really the problem, but of course better refactoring abilities are good. Alexander On Mon, Apr 29, 2013 at 9:59 AM, Niklas Hambüchen m...@nh2.me wrote: Hello Malcolm, no, I had indeed not seen this! Thanks for the link. It goes very much in the direction I was thinking of, but it does not seem to maintained and does not cabal install either. It also seems very much focused on interactive editor integration as compared to written-out transformations. Do you know to what extent they have built and a modification-friendly AST? Also, do you know if the people involved in this are still active in the community and interested in working further in this direction? Thanks Niklas On 29/04/13 15:36, Malcolm Wallace wrote: On 29 Apr 2013, at 07:00, Niklas Hambüchen wrote: I would like to propose the development of source code refactoring tool that operates on Haskell source code ASTs and lets you formulate rewrite rules written in Haskell. Seen this? http://www.haskell.org/haskellwiki/HaRe Regards, Malcolm ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe ___ Haskell-Cafe mailing list Haskell-Cafe@haskell.org http://www.haskell.org/mailman/listinfo/haskell-cafe
Re: [Haskell-cafe] GSoC proposal: Haskell AST-based refactoring and API upgrading tool
Hi Niklas,
I haven't read the whole proposal as I'm short of time. But Alan
Zimmerman is doing a lot of work on integrating HaRe with the GHC API
[1]. He is alanz on freenode and a regular in #hspec.
I haven't looked at the code, but maybe it's of interest to you.
Cheers,
Simon
[1] https://github.com/alanz/HaRe/tree/ghc-api
On Mon, Apr 29, 2013 at 02:00:23PM +0800, Niklas Hambüchen wrote:
I would like to propose the development of source code refactoring tool
that operates on Haskell source code ASTs and lets you formulate rewrite
rules written in Haskell.
Objective
-
The goal is to make refactorings easier and allow global code changes
that might be incredibly tedious to do in a non-automated way.
By making these transformations convenient, we can make it easier to
maintain clean code, add new features or clean up leftovers faster, and
reduce the fear and effort to upgrade to newer versions of packages and
APIs.
Transformations
---
First, here are a few operations you would use this tool for. Some of
them are common operations you would also do in other programming
languages, some are more specific to Haskell.
* Changing all occurrences of import Prelude hiding (catch) to import
qualified Control.Exception as E
* Replacing all uses of a function with that function being imported
qualified or the other way around
* Adding a field to data constructor a record, setting user-supplied
defaults for construction and destruction:
-- Suppose you want to change one of these
data User = User { name :: String, age :: Int }
data User = User String Int
-- into one of these
data User = User { name :: String, age :: Int, active :: Bool }
data User = User String Int Bool
-- the refactoring tool could perform, in all relevant locations:
show (User name age) = ...
show (User name age _) = ...
-- and also this transformation:
... u { name = deleted } ...
... u { name = deleted, active = False } ...
-- or equivalently with records.
-- Special cases could be taken care of as specified, such as
-- whenever an object of [this User type] has
--of its records passed into some function 'email', do this
--now only if the user is active, so modify all relevant code
--email (name u)
--to
--if (active u) then email (name u) else return ()
-- Other examples include adding a position counter to attoparsec.
* Adding a type parameter to a type
-- This happens a lot on monad transformer stacks, e.g.
newtype MyMonad a b c = MyMonad (ReaderT a (WriterT b ...
-- and as you would probably agree on, this is not the most
-- comfortable change to make; in big project this can mean
-- hour-long grinding.
-- It has also recently happened in the basic underlying types
-- of packages like conduit and pipes.
* Adding a new transformer around a monad
* Addressing problems like mentioned in
http://blog.ezyang.com/2012/01/modelling-io/:
There is one last problem with this approach: once the primitives
have been selected, huge swaths of the standard library have to be
redefined by “copy pasting” their definitions ...
* Extracting a value into a let or where clause
* Renaming a variable, and all its occurrences that are semantically
same variable (based on its scope)
* Changing the way things are done, such as:
* Replacing uses of fmap with $, also taking care of the
corresponding import, and such cases were partial application
is involved
* Replacing uses of when (isJust) to forM_
* Making imports clearer by adding all functions used to the file to the
import list of the module that gets them in scope
* Finding all places where an exported function does not have all its
arguments haddock-documented.
* Performing whole-project refactorings instead of operating on single
files only, allowing operations like
Find me all functions of this type, e.g.
Maybe a - (a - m a) - m a
in the project and extract them into this new module,
with the name 'onJust'.
Some of the problems above can be tried to address using regex-based
search and replace, but this already fails in the simplest case of
import Prelude hiding (catch) in case there is more than that imported
from Prelude or newlines involved in the import list.
Transformation on the AST are much more powerful, and can guarantee that
the result is, at least syntactically, valid. No text base tool can do that.
Other uses
--
In addition to being able to perform transformations as mentioned above,
the refactoring tool as a library can be leveraged to:
* Support or be the base of code formatting tools such as
haskell-stylish, linters, style/convention checkers, static analyzers,
test coverage tools etc.
* Implement automatic API upgrades.
Re: [Haskell-cafe] GSoC proposal: Haskell AST-based refactoring and API upgrading tool
The latest updates on HaRe with GHC API project seem to be posted on the
google+ community page:
https://plus.google.com/communities/116266567145785623821
On Monday, April 29, 2013 5:09:56 AM UTC-5, Simon Hengel wrote:
Hi Niklas,
I haven't read the whole proposal as I'm short of time. But Alan
Zimmerman is doing a lot of work on integrating HaRe with the GHC API
[1]. He is alanz on freenode and a regular in #hspec.
I haven't looked at the code, but maybe it's of interest to you.
Cheers,
Simon
[1] https://github.com/alanz/HaRe/tree/ghc-api
On Mon, Apr 29, 2013 at 02:00:23PM +0800, Niklas Hambüchen wrote:
I would like to propose the development of source code refactoring tool
that operates on Haskell source code ASTs and lets you formulate rewrite
rules written in Haskell.
Objective
-
The goal is to make refactorings easier and allow global code changes
that might be incredibly tedious to do in a non-automated way.
By making these transformations convenient, we can make it easier to
maintain clean code, add new features or clean up leftovers faster, and
reduce the fear and effort to upgrade to newer versions of packages and
APIs.
Transformations
---
First, here are a few operations you would use this tool for. Some of
them are common operations you would also do in other programming
languages, some are more specific to Haskell.
* Changing all occurrences of import Prelude hiding (catch) to import
qualified Control.Exception as E
* Replacing all uses of a function with that function being imported
qualified or the other way around
* Adding a field to data constructor a record, setting user-supplied
defaults for construction and destruction:
-- Suppose you want to change one of these
data User = User { name :: String, age :: Int }
data User = User String Int
-- into one of these
data User = User { name :: String, age :: Int, active :: Bool }
data User = User String Int Bool
-- the refactoring tool could perform, in all relevant locations:
show (User name age) = ...
show (User name age _) = ...
-- and also this transformation:
... u { name = deleted } ...
... u { name = deleted, active = False } ...
-- or equivalently with records.
-- Special cases could be taken care of as specified, such as
-- whenever an object of [this User type] has
--of its records passed into some function 'email', do this
--now only if the user is active, so modify all relevant code
--email (name u)
--to
--if (active u) then email (name u) else return ()
-- Other examples include adding a position counter to attoparsec.
* Adding a type parameter to a type
-- This happens a lot on monad transformer stacks, e.g.
newtype MyMonad a b c = MyMonad (ReaderT a (WriterT b ...
-- and as you would probably agree on, this is not the most
-- comfortable change to make; in big project this can mean
-- hour-long grinding.
-- It has also recently happened in the basic underlying types
-- of packages like conduit and pipes.
* Adding a new transformer around a monad
* Addressing problems like mentioned in
http://blog.ezyang.com/2012/01/modelling-io/:
There is one last problem with this approach: once the primitives
have been selected, huge swaths of the standard library have to be
redefined by “copy pasting” their definitions ...
* Extracting a value into a let or where clause
* Renaming a variable, and all its occurrences that are semantically
same variable (based on its scope)
* Changing the way things are done, such as:
* Replacing uses of fmap with $, also taking care of the
corresponding import, and such cases were partial application
is involved
* Replacing uses of when (isJust) to forM_
* Making imports clearer by adding all functions used to the file to the
import list of the module that gets them in scope
* Finding all places where an exported function does not have all its
arguments haddock-documented.
* Performing whole-project refactorings instead of operating on single
files only, allowing operations like
Find me all functions of this type, e.g.
Maybe a - (a - m a) - m a
in the project and extract them into this new module,
with the name 'onJust'.
Some of the problems above can be tried to address using regex-based
search and replace, but this already fails in the simplest case of
import Prelude hiding (catch) in case there is more than that imported
from Prelude or newlines involved in the import list.
Transformation on the AST are much more powerful, and can guarantee that
the result is, at least syntactically, valid. No text base tool can do
that.
