jorisvandenbossche commented on PR #51680:
URL: https://github.com/apache/arrow/pull/51680#issuecomment-6016178130
> Yes, kernels are never available statically. Can we technically put stub
information into .py files? Seems unconventional.
Well, that's called type annotations? ;) I mean, whether a function is
defined statically or defined by code, doesn't really matter for python (of
course it matters to static analyzers). Just as we currently add a name,
module, signature, etc to the generated function, we can also add a signature
(in the compute.py source code).
That won't be useful for static analyzers (just as those currently also
cannot even see that those functions exist), but generating a stub file from
the dynamic python functions should be possible?
> > I think we could relatively easily also generate those [compute.py]
stubs? (we already generate the python code with signature on the fly, then we
can also generate annotations on the fly and save that with some extra script
to a file)
>
> Ok, but we can't have rich types. I don't think we want that.
Again as a non-typing expert, can you give a bit more details why that is
not possible? Those rich types cannot be used as python variables? (i.e. you
cannot write them as `wrapper.__annotations__ = {"arg": ...}`?)
> > Then we can still only provide a stub file for compute.py, and do inline
annotations for the other python files?
>
> Yes, and we have two separate ways and locations for storing annotations.
I am not against this, but I'd note it complicates things.
AFAIK that is quite standard practice for libraries with compiled modules?
Stub files for the compiled modules, and inline annotations for the python
modules? (at least that is what pandas does, but I haven't been involved myself
in setting that up) I do see that eg numpy also uses stub files for the python
files.
If we would use inline annotations for python files in general, then we
would have two "ways" anyhow. In that case `compute.py` would just be an
exception on the rule (having a stub instead of inline annotations), rather
than adding a separate way to store annotations.
> Found the thread [#45919 (reply in
thread)](https://github.com/apache/arrow/discussions/45919#discussioncomment-14275898)
That thread is from a context discussing the manual stubs, though, were
including docstrings means they have to be kept up to date manually. If we
auto-generate stubs, I think that is a different situation.
Reading that thread, I like Marco's comment at
https://github.com/apache/arrow/discussions/45919#discussioncomment-14297417,
though, quoting: "start simple". Whether we go with auto-generated stubs (as
discussed here) or with the manual stubs approach, I would highly recommend
starting simple (and the current PRs are far from that).
> I think the stalled PRs is mostly on me not having full time availability
for this.
That is certainly a large part of it (and not only your time, but time of
anyone who would want to move this forward). But those types being quite
complicated also does not help. PRs adding simple type annotations could be
reviewed / merged a lot more easily.
> As for the proposed types - do you believe they are overly complex for
what's needed downstream?
I don't know exactly what is needed downstream, so I cannot comment on
"overly complex _for what's needed_", I can only say that _I_ as a maintainer
find them very complex.
And I also know that for a lot of downstream usage, already having basic
types would be a gigantic step forward from having no types. For example, to
enable a large part of tab completion / method calls evaluation in IDEs, just
having basic return types would (AFAIU) already do a lot (e.g. the fact that an
IDE can know that the result of `pa.table()` is a `pa.Table` and which methods
are available for such a table, and for each of those methods what the accepted
arguments are and the return type, etc)
> Let's try to set a target type complexity and then find a way to it. From
memory types are pretty close to what you need if you want to e.g. infer what
will be output type of a function given an input type. If this is something we
ultimately want I don't think we can simplify much.
If we now throw the internal types away and start with simple types we'll
have to recreate internal types later to get to desired functionality which I
would like to avoid.
I still think that even if we want to have such powerful type annotations
such that analyzers can infer _pyarrow types_ eventually, we should start with
simple types (to ensure we provide most of the value on a shorter term). That
might indeed mean multiple iterations on the types, and potentially changing
approach along the way. But I think the end result (and the value of the
intermediate steps) will be better.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
