ject: Re: Spark 1.4.0 pyspark and pylint breaking
I think relative imports can not help in this case.
When you run scripts in pyspark/sql, it doesn't know anything about
pyspark.sql, it
just see types.py as a separate module.
On Tue, May 26, 2015 at 12:44 PM, Punyashloka Biswal
wrote:
> Davi
I think relative imports can not help in this case.
When you run scripts in pyspark/sql, it doesn't know anything about
pyspark.sql, it
just see types.py as a separate module.
On Tue, May 26, 2015 at 12:44 PM, Punyashloka Biswal
wrote:
> Davies: Can we use relative imports (import .types) in the
When you run the test in python/pyspark/sql/ by
bin/spark-submit python/pyspark/sql/dataframe.py
the the current directory is the first item in sys.path, sql/types.py
will have higher priority then python3.4/types.py, the tests will
fail.
On Tue, May 26, 2015 at 12:08 PM, Justin Uang wrote:
> T
Davies: Can we use relative imports (import .types) in the unit tests in
order to disambiguate between the global and local module?
Punya
On Tue, May 26, 2015 at 3:09 PM Justin Uang wrote:
> Thanks for clarifying! I don't understand python package and modules names
> that well, but I thought th
Thanks for clarifying! I don't understand python package and modules names
that well, but I thought that the package namespacing would've helped,
since you are in pyspark.sql.types. I guess not?
On Tue, May 26, 2015 at 3:03 PM Davies Liu wrote:
> There is a module called 'types' in python 3:
>
>
There is a module called 'types' in python 3:
davies@localhost:~/work/spark$ python3
Python 3.4.1 (v3.4.1:c0e311e010fc, May 18 2014, 00:54:21)
[GCC 4.2.1 (Apple Inc. build 5666) (dot 3)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import types
>>> types
W
In commit 04e44b37, the migration to Python 3, pyspark/sql/types.py was
renamed to pyspark/sql/_types.py and then some magic in
pyspark/sql/__init__.py dynamically renamed the module back to types. I
imagine that this is some naming conflict with Python 3, but what was the
error that showed up?
Th
