[jira] [Commented] (SPARK-36845) Inline type hint files
[
https://issues.apache.org/jira/browse/SPARK-36845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17432755#comment-17432755
]
dch nguyen commented on SPARK-36845:
[~ueshin], yes, i will
> Inline type hint files
> --
>
> Key: SPARK-36845
> URL: https://issues.apache.org/jira/browse/SPARK-36845
> Project: Spark
> Issue Type: Umbrella
> Components: PySpark, SQL
>Affects Versions: 3.3.0
>Reporter: Takuya Ueshin
>Assignee: Xinrong Meng
>Priority: Major
>
> Currently there are type hint stub files ({{*.pyi}}) to show the expected
> types for functions, but we can also take advantage of static type checking
> within the functions by inlining the type hints.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36845) Inline type hint files
[
https://issues.apache.org/jira/browse/SPARK-36845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17432684#comment-17432684
]
Takuya Ueshin commented on SPARK-36845:
---
Hi [~dchvn], shall we file separate umbrella tickets for each module and
resolve this?
The number of sub tasks are already growing for one umbrella ticket.
Managing tasks based on each module should be clearer.
Thanks!
> Inline type hint files
> --
>
> Key: SPARK-36845
> URL: https://issues.apache.org/jira/browse/SPARK-36845
> Project: Spark
> Issue Type: Umbrella
> Components: PySpark, SQL
>Affects Versions: 3.3.0
>Reporter: Takuya Ueshin
>Assignee: Xinrong Meng
>Priority: Major
>
> Currently there are type hint stub files ({{*.pyi}}) to show the expected
> types for functions, but we can also take advantage of static type checking
> within the functions by inlining the type hints.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36845) Inline type hint files
[
https://issues.apache.org/jira/browse/SPARK-36845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17422989#comment-17422989
]
Maciej Szymkiewicz commented on SPARK-36845:
It is an optional library and won't be present by default.
As an official PSF project without transitive dependencies it might be
acceptable as an installation dependency for PySpark, but it still leaves
{{pyspark}} shell.
It might be possible to use {{TYPE_CHECKING}} blocks and {{__future__
annotations}} like this:
{code:python}
# _typing.py
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from typing import Protocol
class P(Protocol): ...
{code}
{code:python}
from __future__ import annotations
from typing import TYPE_CHECKING
if TYPE_CHECKING:
from _typing import P
def f(p: P):
pass
{code}
but it requires further testing and might be not be worth it after all.
> Inline type hint files
> --
>
> Key: SPARK-36845
> URL: https://issues.apache.org/jira/browse/SPARK-36845
> Project: Spark
> Issue Type: Umbrella
> Components: PySpark, SQL
>Affects Versions: 3.3.0
>Reporter: Takuya Ueshin
>Priority: Major
>
> Currently there are type hint stub files ({{*.pyi}}) to show the expected
> types for functions, but we can also take advantage of static type checking
> within the functions by inlining the type hints.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36845) Inline type hint files
[
https://issues.apache.org/jira/browse/SPARK-36845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17422935#comment-17422935
]
Takuya Ueshin commented on SPARK-36845:
---
Is the {{typing-extensions}} always available in runtime Python 3.7
environment? or do users need to install some libraries?
> Inline type hint files
> --
>
> Key: SPARK-36845
> URL: https://issues.apache.org/jira/browse/SPARK-36845
> Project: Spark
> Issue Type: Umbrella
> Components: PySpark, SQL
>Affects Versions: 3.3.0
>Reporter: Takuya Ueshin
>Priority: Major
>
> Currently there are type hint stub files ({{*.pyi}}) to show the expected
> types for functions, but we can also take advantage of static type checking
> within the functions by inlining the type hints.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36845) Inline type hint files
[
https://issues.apache.org/jira/browse/SPARK-36845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17422576#comment-17422576
]
Maciej Szymkiewicz commented on SPARK-36845:
{quote} For the {{_typing.pyi}}, it uses {{Protocol}} that is supported Python
3.8 and above, so it won't be a straightforward, IIUC
{quote}
We still use typing-extensions for {{Protocol}}, so that shouldn't be a problem.
> Inline type hint files
> --
>
> Key: SPARK-36845
> URL: https://issues.apache.org/jira/browse/SPARK-36845
> Project: Spark
> Issue Type: Umbrella
> Components: PySpark, SQL
>Affects Versions: 3.3.0
>Reporter: Takuya Ueshin
>Priority: Major
>
> Currently there are type hint stub files ({{*.pyi}}) to show the expected
> types for functions, but we can also take advantage of static type checking
> within the functions by inlining the type hints.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36845) Inline type hint files
[
https://issues.apache.org/jira/browse/SPARK-36845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17422487#comment-17422487
]
Takuya Ueshin commented on SPARK-36845:
---
For the {{_typing.pyi}}, it uses {{Protocol}} that is supported Python 3.8 and
above, so it won't be a straightforward, IIUC.
> Inline type hint files
> --
>
> Key: SPARK-36845
> URL: https://issues.apache.org/jira/browse/SPARK-36845
> Project: Spark
> Issue Type: Umbrella
> Components: PySpark, SQL
>Affects Versions: 3.3.0
>Reporter: Takuya Ueshin
>Priority: Major
>
> Currently there are type hint stub files ({{*.pyi}}) to show the expected
> types for functions, but we can also take advantage of static type checking
> within the functions by inlining the type hints.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36845) Inline type hint files
[
https://issues.apache.org/jira/browse/SPARK-36845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17422486#comment-17422486
]
Takuya Ueshin commented on SPARK-36845:
---
It would be great if we could use the "annotation" future flag!
> Inline type hint files
> --
>
> Key: SPARK-36845
> URL: https://issues.apache.org/jira/browse/SPARK-36845
> Project: Spark
> Issue Type: Umbrella
> Components: PySpark, SQL
>Affects Versions: 3.3.0
>Reporter: Takuya Ueshin
>Priority: Major
>
> Currently there are type hint stub files ({{*.pyi}}) to show the expected
> types for functions, but we can also take advantage of static type checking
> within the functions by inlining the type hints.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36845) Inline type hint files
[
https://issues.apache.org/jira/browse/SPARK-36845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17422464#comment-17422464
]
Hyukjin Kwon commented on SPARK-36845:
--
Yeah, _typing.pyi one I am not sure. Agree we should think about which approach
we'll take on this ..
I added SPARK-36145 as a related ticket for now. I will make sure making it
resolved all together for Spark 3.3.
> Inline type hint files
> --
>
> Key: SPARK-36845
> URL: https://issues.apache.org/jira/browse/SPARK-36845
> Project: Spark
> Issue Type: Umbrella
> Components: PySpark, SQL
>Affects Versions: 3.3.0
>Reporter: Takuya Ueshin
>Priority: Major
>
> Currently there are type hint stub files ({{*.pyi}}) to show the expected
> types for functions, but we can also take advantage of static type checking
> within the functions by inlining the type hints.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36845) Inline type hint files
[
https://issues.apache.org/jira/browse/SPARK-36845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17422379#comment-17422379
]
Maciej Szymkiewicz commented on SPARK-36845:
Additionally, we should probably rethink `_typing.pyi` modules. In general it
should be safe to move these to plain Python modules (see for example how
Pandas folks do similar thing).
> Inline type hint files
> --
>
> Key: SPARK-36845
> URL: https://issues.apache.org/jira/browse/SPARK-36845
> Project: Spark
> Issue Type: Umbrella
> Components: PySpark, SQL
>Affects Versions: 3.3.0
>Reporter: Takuya Ueshin
>Priority: Major
>
> Currently there are type hint stub files ({{*.pyi}}) to show the expected
> types for functions, but we can also take advantage of static type checking
> within the functions by inlining the type hints.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36845) Inline type hint files
[
https://issues.apache.org/jira/browse/SPARK-36845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17422371#comment-17422371
]
Maciej Szymkiewicz commented on SPARK-36845:
I'd recommend adding SPARK-36145 as blocker for this, so we can proceed cleanly
with {{from __future__ import annotations}} and avoid all the quoting in
inlined annotations.
> Inline type hint files
> --
>
> Key: SPARK-36845
> URL: https://issues.apache.org/jira/browse/SPARK-36845
> Project: Spark
> Issue Type: Umbrella
> Components: PySpark, SQL
>Affects Versions: 3.3.0
>Reporter: Takuya Ueshin
>Priority: Major
>
> Currently there are type hint stub files ({{*.pyi}}) to show the expected
> types for functions, but we can also take advantage of static type checking
> within the functions by inlining the type hints.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36845) Inline type hint files
[
https://issues.apache.org/jira/browse/SPARK-36845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17421865#comment-17421865
]
Hyukjin Kwon commented on SPARK-36845:
--
please go ahead.
> Inline type hint files
> --
>
> Key: SPARK-36845
> URL: https://issues.apache.org/jira/browse/SPARK-36845
> Project: Spark
> Issue Type: Umbrella
> Components: PySpark, SQL
>Affects Versions: 3.3.0
>Reporter: Takuya Ueshin
>Priority: Major
>
> Currently there are type hint stub files ({{*.pyi}}) to show the expected
> types for functions, but we can also take advantage of static type checking
> within the functions by inlining the type hints.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-36845) Inline type hint files
[
https://issues.apache.org/jira/browse/SPARK-36845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17421859#comment-17421859
]
dgd_contributor commented on SPARK-36845:
-
Can i work on this :D ?
> Inline type hint files
> --
>
> Key: SPARK-36845
> URL: https://issues.apache.org/jira/browse/SPARK-36845
> Project: Spark
> Issue Type: Umbrella
> Components: PySpark, SQL
>Affects Versions: 3.3.0
>Reporter: Takuya Ueshin
>Priority: Major
>
> Currently there are type hint stub files ({{*.pyi}}) to show the expected
> types for functions, but we can also take advantage of static type checking
> within the functions by inlining the type hints.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
