[
https://issues.apache.org/jira/browse/SPARK-26591?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16742517#comment-16742517
]
Bryan Cutler commented on SPARK-26591:
--------------------------------------
I created the same virtual environment and could not reproduce, can anyone else
verify?
OS: 4.15.0-43-generic #46~16.04.1-Ubuntu SMP Fri Dec 7 13:31:08 UTC 2018
x86_64 x86_64 x86_64 GNU/Linux
{noformat}
Welcome to
____ __
/ __/__ ___ _____/ /__
_\ \/ _ \/ _ `/ __/ '_/
/__ / .__/\_,_/_/ /_/\_\ version 2.4.0
/_/
Using Python version 3.6.7 (default, Nov 21 2018 02:32:25)
SparkSession available as 'spark'.
>>> import pyarrow
>>> pyarrow.__version__
'0.11.1'
>>> import pandas
>>> pandas.__version__
'0.23.4'
>>> import numpy
>>> numpy.__version__
'1.15.4'
>>> from pyspark.sql.functions import pandas_udf, PandasUDFType
>>> from pyspark.sql.types import IntegerType, StringType
>>> slen = pandas_udf(lambda s: s.str.len(), IntegerType())
>>> slen
<function <lambda> at 0x7f099a99cd90>
{noformat}
> Scalar Pandas UDF fails with 'illegal hardware instruction' in a certain
> environment
> ------------------------------------------------------------------------------------
>
> Key: SPARK-26591
> URL: https://issues.apache.org/jira/browse/SPARK-26591
> Project: Spark
> Issue Type: Bug
> Components: PySpark
> Affects Versions: 2.4.0
> Environment: Python 3.6.7
> Pyspark 2.4.0
> OS:
> {noformat}
> Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018 x86_64
> x86_64 x86_64 GNU/Linux{noformat}
> CPU:
>
> {code:java}
> Dual core AMD Athlon II P360 (-MCP-) cache: 1024 KB
> clock speeds: max: 2300 MHz 1: 1700 MHz 2: 1700 MHz
> {code}
>
>
> Reporter: Elchin
> Priority: Major
> Attachments: core
>
>
> When I try to use pandas_udf from examples in
> [documentation|https://spark.apache.org/docs/2.4.0/api/python/pyspark.sql.html#pyspark.sql.functions.pandas_udf]:
> {code:java}
> from pyspark.sql.functions import pandas_udf, PandasUDFType
> from pyspark.sql.types import IntegerType, StringType
> slen = pandas_udf(lambda s: s.str.len(), IntegerType()) #here it is
> crashed{code}
> I get the error:
> {code:java}
> [1] 17969 illegal hardware instruction (core dumped) python3{code}
> The environment is:
> Python 3.6.7
> PySpark 2.4.0
> PyArrow: 0.11.1
> Pandas: 0.23.4
> NumPy: 1.15.4
> OS: Linux 4.15.0-43-generic #46-Ubuntu SMP Thu Dec 6 14:45:28 UTC 2018
> x86_64 x86_64 x86_64 GNU/Linux
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
