I need to be able to tell the types of fields, in particular are fields of
a struct a native type or a struct themselves.
The ast parse even with a simple importer don’t provide that info.
On Sat, 16 Oct 2021 at 21:06, 'Richard Oudkerk' via golang-nuts <
golang-nuts@googlegroups.com> wrote:
> I am not sure what "import external packages" means.
>
> Apart dot imports (which I have never seen used for real) why would you
> need to load the imported packages?
>
> On Saturday, 16 October 2021 at 20:34:17 UTC+1 Steven Hartland wrote:
>
>> Thanks Richard, that allowed me to replace a hand rolled universe scope 👍
>>
>> My importer varies from yours in that for correct lookups for versioned
>> packages or those with '-' in I had to copy ImportPathToAssumedName from
>> x/tools/internal/imports/fix.go.
>>
>> func simpleImporter(imports map[string]*ast.Object, path string)
>> (*ast.Object, error) {
>> pkg := imports[path]
>> if pkg == nil {
>> pkg = ast.NewObj(ast.Pkg, ImportPathToAssumedName(path))
>> pkg.Data = ast.NewScope(nil) // required by
>> ast.NewPackage for dot-import
>> imports[path] = pkg
>> }
>> return pkg, nil
>> }
>>
>> This now works for all cases which don't import external packages. So now
>> I just need to do the on demand load of packages, which I suspect will lead
>> me right back to packages.Load.
>>
>> On Sat, 16 Oct 2021 at 15:59, 'Richard Oudkerk' via golang-nuts <
>> golan...@googlegroups.com> wrote:
>>
>>> You could try building the universe scope for ast.NewPackage from
>>> types.Universe. For example
>>>
>>> https://play.golang.org/p/1E5Iu4vW3g9
>>>
>>> func NewPackage(fset *token.FileSet, files map[string]*ast.File)
>>> (*ast.Package, error) {
>>> univ, err := universe()
>>> if err != nil {
>>> return nil, err
>>> }
>>> return ast.NewPackage(fset, files, dummyImporter, univ)
>>> }
>>>
>>> func dummyImporter(imports map[string]*ast.Object, importPath string)
>>> (*ast.Object, error) {
>>> pkg := imports[importPath]
>>> if pkg == nil {
>>> pkg = ast.NewObj(ast.Pkg, path.Base(importPath))
>>> pkg.Data = ast.NewScope(nil)
>>> imports[importPath] = pkg
>>> }
>>> return pkg, nil
>>> }
>>>
>>> func universe() (*ast.Scope, error) {
>>> u := ast.NewScope(nil)
>>> for _, name := range types.Universe.Names() {
>>> o := types.Universe.Lookup(name)
>>> if o == nil {
>>> return nil, fmt.Errorf("failed to lookup %s in universe scope", name)
>>> }
>>> var objKind ast.ObjKind
>>> switch o.(type) {
>>> case *types.Const, *types.Nil:
>>> objKind = ast.Con
>>> case *types.TypeName:
>>> objKind = ast.Typ
>>> case *types.Builtin:
>>> objKind = ast.Fun
>>> default:
>>> return nil, fmt.Errorf("unexpected builtin %s of type %T", o.Name(), o)
>>> }
>>> obj := ast.NewObj(objKind, name)
>>> if u.Insert(obj) != nil {
>>> return nil, fmt.Errorf("types internal error: double declaration")
>>> }
>>> obj.Decl = u
>>> }
>>> return u, nil
>>> }
>>>
>>> On Saturday, 16 October 2021 at 14:38:43 UTC+1 eli...@gmail.com wrote:
>>>
>>>> On Fri, Oct 15, 2021 at 2:13 PM Steven Hartland <ste...@multiplay.co.uk>
>>>> wrote:
>>>>
>>>>> I converted my code to x/tools/go/packages
>>>>> <https://pkg.go.dev/golang.org/x/tools@v0.1.7/go/packages> and while
>>>>> it did solve the problem it's VERY slow in comparison.
>>>>>
>>>>> I have a set of 21 tests operating on a single package which has at
>>>>> most two very basic types, no imports and using go/parser
>>>>> <https://pkg.go.dev/go/parser> they take 0.011s but with go/packages
>>>>> <https://pkg.go.dev/golang.org/x/tools@v0.1.7/go/packages> that
>>>>> increases to 3.548s a 300x slow down.
>>>>>
>>>>> I'm setting a basic mode: packages.NeedName | packages.NeedSyntax
>>>>>
>>>>> The package.Load call takes ~220ms whereas ast.NewPackage only
>>>>> takes 2.7µs.
>>>>>
>>>>
>>>> Could you post a reproducer of your target package and analysis
>>>> somewhere? 220ms for packages.Load sounds like a lot. It's true that
>>>> packages does a lot more work than just the parser (*), but it's not
>>>> supposed to be that slow. In my tests a simple Load with more functionality
>>>> takes 60-70ms
>>>>
>>>> (*) The type checking takes a bit of time over just parsing to AST, but
>>>> the biggest difference is loading multiple files from imports. For type
>>>> checking you need to know, when you see:
>>>>
>>>> import foo
>>>>
>>>> x := foo.Foo()
>>>>
>>>> What the type of `x` is, so go/packages has to analyze the `foo`
>>>> package as well.
>>>>
>>>>
>>>>
>>>>>
>>>>> As the resulting ast.File's are pretty much the same, I'm wondering if
>>>>> for my use case packages.Load is doing way more than I need?
>>>>>
>>>>> Another downside is for tests run in a temporary directory outside of
>>>>> the package space package.Load fails with:
>>>>> directory /tmp/tests76985775 outside available modules
>>>>>
>>>>> I fixed it by calling ioutil.TempDir with "." but that's not ideal.
>>>>>
>>>>> Thoughts?
>>>>>
>>>>> On Tue, 12 Oct 2021 at 13:42, Steven Hartland <ste...@multiplay.co.uk>
>>>>> wrote:
>>>>>
>>>>>> Thanks David, much appreciated, I will have a look at both.
>>>>>>
>>>>>> When migrating from go/ast to go/types did you hit anything of note
>>>>>> I should look out for?
>>>>>>
>>>>>> On Mon, 11 Oct 2021 at 17:06, David Finkel <david....@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Oct 11, 2021 at 5:48 AM Steven Hartland <
>>>>>>> ste...@multiplay.co.uk> wrote:
>>>>>>>
>>>>>>>> If the ast.Files passed to ast.NewPackage includes built in types
>>>>>>>> such as int it returns an error e.g.
>>>>>>>> file1.go:5:6: undeclared name: int
>>>>>>>>
>>>>>>>> Is there a way to prevent that?
>>>>>>>>
>>>>>>>
>>>>>>> Generally, I always add the `builtin` package to the list of
>>>>>>> packages I'm parsing.
>>>>>>> I wrote a little library for exactly this kind of package loading a
>>>>>>> few years ago:
>>>>>>> https://gitlab.com/dfinkel/goastpkg/-/blob/master/go_ast_parser.go
>>>>>>> (https://pkg.go.dev/golang.spin-2.net/astpkg)
>>>>>>>
>>>>>>>>
>>>>>>>> Playground example: https://play.golang.org/p/Yg30TTzoLHP
>>>>>>>>
>>>>>>>> My goal is to take multiple files, resolve inter file dependencies
>>>>>>>> e.g. a type referencing another type in a different file and process
>>>>>>>> the
>>>>>>>> resulting ast.Files. So if there is a better way to achieve this I'm
>>>>>>>> all
>>>>>>>> ears.
>>>>>>>>
>>>>>>>
>>>>>>> In general, I've stopped using the `go/ast` internal references as
>>>>>>> much and have started using resolved `go/types` references as they're
>>>>>>> more
>>>>>>> reliable and better-specified.
>>>>>>> (golang.org/x/tools/go/packages
>>>>>>> <https://pkg.go.dev/golang.org/x/tools@v0.1.7/go/packages> has a
>>>>>>> LoadMode flag for generating `go/types.Info` (NeedTypesInfo
>>>>>>> <https://pkg.go.dev/golang.org/x/tools@v0.1.7/go/packages#NeedTypesInfo>
>>>>>>> ))
>>>>>>>
>>>>>>>>
>>>>>>>> Regards
>>>>>>>> Steve
>>>>>>>>
>>>>>>>> --
>>>>>>>> You received this message because you are subscribed to the Google
>>>>>>>> Groups "golang-nuts" group.
>>>>>>>> To unsubscribe from this group and stop receiving emails from it,
>>>>>>>> send an email to golang-nuts...@googlegroups.com.
>>>>>>>> To view this discussion on the web visit
>>>>>>>> https://groups.google.com/d/msgid/golang-nuts/CAHEMsqbJoJxuo3c-mofMtzXXJhYCzV2skW2ZB3ZPY6WtA8%2BxHw%40mail.gmail.com
>>>>>>>> <https://groups.google.com/d/msgid/golang-nuts/CAHEMsqbJoJxuo3c-mofMtzXXJhYCzV2skW2ZB3ZPY6WtA8%2BxHw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>>>>> .
>>>>>>>>
>>>>>>> --
>>>>> You received this message because you are subscribed to the Google
>>>>> Groups "golang-nuts" group.
>>>>> To unsubscribe from this group and stop receiving emails from it, send
>>>>> an email to golang-nuts...@googlegroups.com.
>>>>>
>>>> To view this discussion on the web visit
>>>>> https://groups.google.com/d/msgid/golang-nuts/CAHEMsqYMSBUfuOUvptv6UrvBFTwFxjOhJZ5sMN-omOx5ESL5hw%40mail.gmail.com
>>>>> <https://groups.google.com/d/msgid/golang-nuts/CAHEMsqYMSBUfuOUvptv6UrvBFTwFxjOhJZ5sMN-omOx5ESL5hw%40mail.gmail.com?utm_medium=email&utm_source=footer>
>>>>> .
>>>>
>>>>
>>>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "golang-nuts" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to golang-nuts...@googlegroups.com.
>>>
>> To view this discussion on the web visit
>>> https://groups.google.com/d/msgid/golang-nuts/d570a7ce-a780-46d8-a323-f9c26a6c2561n%40googlegroups.com
>>> <https://groups.google.com/d/msgid/golang-nuts/d570a7ce-a780-46d8-a323-f9c26a6c2561n%40googlegroups.com?utm_medium=email&utm_source=footer>
>>> .
>>>
>> --
> You received this message because you are subscribed to the Google Groups
> "golang-nuts" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to golang-nuts+unsubscr...@googlegroups.com.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/golang-nuts/6aaa7c3a-7ef5-47ea-9f29-75443a4599b6n%40googlegroups.com
> <https://groups.google.com/d/msgid/golang-nuts/6aaa7c3a-7ef5-47ea-9f29-75443a4599b6n%40googlegroups.com?utm_medium=email&utm_source=footer>
> .
>
--
You received this message because you are subscribed to the Google Groups
"golang-nuts" group.
To unsubscribe from this group and stop receiving emails from it, send an email
to golang-nuts+unsubscr...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/golang-nuts/CAHEMsqaQ9%3Dia2MXrVEXt--Qk3Nrx1UXK2JYU0D2BxBcZX%2B5mxw%40mail.gmail.com.
