Ganesh Kumar created SPARK-28567:
------------------------------------
Summary: pyspark-sql returns error "Paths and fileStatuses are
expected to be 1. . Got paths: 1 fileStatuses: 26"
Key: SPARK-28567
URL: https://issues.apache.org/jira/browse/SPARK-28567
Project: Spark
Issue Type: Bug
Components: PySpark, SQL
Affects Versions: 2.3.2
Environment: Sql below..
query = "WITH hdr as ( \
select vin_nbr, element_cd, max(element_timstm) as element_timstmp from
dev_edm.gk_veh_element \
where vin_nbr like '2C%' and element_timstm >= cast('{0}' as timestamp) -
interval '1' day \
and element_timstm < cast('{1}' as timestamp) \
and ((yr_nbr=substring('{2}', 1, 4) and mth_nbr=substring('{3}', 6, 2) and
day_nbr=substring('{4}', 9, 2)) \
or (yr_nbr=substring('{5}', 1, 4) and mth_nbr=substring('{6}', 6, 2) and
day_nbr=substring('{7}', 9, 2))) \
and ( \
(ELEMENT_CD = 'LEFT_FR_TIRE_PRESSURE_STATUS_LOW_FLAG' and lower(ELEMENT_VAL) =
'true' ) or \
(ELEMENT_CD = 'RIGHT_FR_TIRE_PRESSURE_STATUS_LOW_FLAG' and lower(ELEMENT_VAL) =
'true') or \
(ELEMENT_CD = 'LEFT_RR_TIRE_PRESSURE_STATUS_LOW_FLAG' and lower(ELEMENT_VAL) =
'true' ) or \
(ELEMENT_CD = 'RIGHT_RR_TIRE_PRESSURE_STATUS_LOW_FLAG' and lower(ELEMENT_VAL) =
'true') or \
(ELEMENT_CD in ('RIGHT_FRONT_TIRE_PRESSURE', 'RIGHT_REAR_TIRE_PRESSURE',
'LEFT_FRONT_TIRE_PRESSURE', 'LEFT_REAR_TIRE_PRESSURE') ) or \
(ELEMENT_CD = 'DIESEL_EXHAUST_FLUID' and (lower(ELEMENT_VAL) = 'on' ) ) or \
(ELEMENT_CD = 'BRAKE_FLUID_LOW' and (lower(ELEMENT_VAL) = 'true' ) ) or \
(ELEMENT_CD = 'BATTERY_CRITICALLY_LOW' and (lower(ELEMENT_VAL) = 'true' ) ) or
\
(ELEMENT_CD = 'ODO_READ' ) or \
(ELEMENT_CD = 'EOL_READ' and ELEMENT_VAL < 30 ) ) \
group by vin_nbr, element_cd \
order by vin_nbr, element_cd) \
SELECT distinct hdr.vin_nbr, hdr.element_cd, \
case when dtl.element_cd like '%PRESSURE' then cast(cast(round(dtl.element_val
* 1000 / 6894.76) as int) as string) else dtl.element_val end element_val, \
hdr.element_timstmp, 'BDC LOAD' load_process_nm, 'CFL_SVC' created_user_id,
current_timestamp created_timstm, 'CFL_SVC' updated_user_id, current_timestamp
updated_timstm \
from dev_edm.gk_veh_element_np dtl, hdr \
where dtl.vin_nbr = hdr.vin_nbr \
and dtl.element_cd = hdr.element_cd \
and dtl.element_timstm = hdr.element_timstmp \
and ((dtl.yr_nbr=substring('{8}', 1, 4) and dtl.mth_nbr=substring('{9}', 6, 2)
and dtl.day_nbr=substring('{10}', 9, 2)) \
or (dtl.yr_nbr=substring('{11}', 1, 4) and dtl.mth_nbr=substring('{12}', 6,
2) and dtl.day_nbr=substring('{11}', 9, 2))) \
and case \
when dtl.element_cd like '%PRESSURE_STATUS_LOW_FLAG' then
lower(dtl.element_val) = 'true' \
when dtl.element_cd = 'DIESEL_EXHAUST_FLUID' then lower(dtl.element_val) =
'on' \
when dtl.element_cd = 'BRAKE_FLUID_LOW' then lower(dtl.element_val) = 'true' \
when dtl.element_cd = 'BATTERY_CRITICALLY_LOW' then lower(dtl.element_val) =
'true' \
else 1=1 end".format(crntdate, crntdate, prevdate, prevdate, prevdate,
crntdate, crntdate, crntdate, prevdate, prevdate, prevdate, crntdate, crntdate,
crntdate );
df = hive.executeQuery(query)
df.show(10)
Reporter: Ganesh Kumar
df.show(10) errors out.
pyspark-sql returns an error " Caused by:
org.apache.hadoop.hive.ql.metadata.HiveException: java.io.IOException:
java.lang.IllegalStateException: Requested to generate single split. Paths and
fileStatuses are expected to be 1. Got paths: 1 fileStatuses: 26"
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
