[
https://issues.apache.org/jira/browse/SPARK-22296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Randy Tidd updated SPARK-22296:
-------------------------------
Description:
This is with Scala 2.11.
We have a case class that has a constructor with 85 args, the last two of which
are:
var chargesInst :
scala.collection.mutable.Seq[ChargeInstitutional] =
scala.collection.mutable.Seq.empty[ChargeInstitutional],
var chargesProf :
scala.collection.mutable.Seq[ChargeProfessional] =
scala.collection.mutable.Seq.empty[ChargeProfessional]
A mutable Seq in a the constructor of a case class is probably poor form but
Scala allows it. When we run this job we get this error:
build 17-Oct-2017 05:30:50 2017-10-17 09:30:50 [Executor task launch
worker-1] ERROR o.a.s.s.c.e.codegen.CodeGenerator - failed to compile:
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line
8217, Column 70: No applicable constructor/method found for actual parameters
"java.lang.String, java.lang.String, long, java.lang.String, long, long, long,
java.lang.String, long, long, double, scala.Option, scala.Option,
java.lang.String, java.lang.String, long, java.lang.String, java.lang.String,
java.lang.String, java.lang.String, java.lang.String, java.lang.String, int,
java.lang.String, java.lang.String, java.lang.String, java.lang.String,
java.lang.String, java.lang.String, java.lang.String, java.lang.String,
java.lang.String, java.lang.String, long, long, long, long, long, scala.Option,
scala.Option, scala.Option, scala.Option, scala.Option, java.lang.String,
java.lang.String, java.lang.String, java.lang.String, java.lang.String,
java.lang.String, long, java.lang.String, int, double, double,
java.lang.String, java.lang.String, java.lang.String, long, java.lang.String,
java.lang.String, java.lang.String, java.lang.String, long, long, long, long,
java.lang.String, com.xyz.xyz.xyz.domain.Patient,
com.xyz.xyz.xyz.domain.Physician, scala.collection.Seq, scala.collection.Seq,
java.lang.String, long, java.lang.String, int, int, boolean, boolean,
scala.collection.Seq, boolean, scala.collection.Seq, boolean,
scala.collection.Seq, scala.collection.Seq"; candidates are:
"com.xyz.xyz.xyz.domain.Account(java.lang.String, java.lang.String, long,
java.lang.String, long, long, long, java.lang.String, long, long, double,
scala.Option, scala.Option, java.lang.String, java.lang.String, long,
java.lang.String, java.lang.String, java.lang.String, java.lang.String,
java.lang.String, java.lang.String, int, java.lang.String, java.lang.String,
java.lang.String, java.lang.String, java.lang.String, java.lang.String,
java.lang.String, java.lang.String, java.lang.String, java.lang.String, long,
long, long, long, long, scala.Option, scala.Option, scala.Option, scala.Option,
scala.Option, java.lang.String, java.lang.String, java.lang.String,
java.lang.String, java.lang.String, java.lang.String, long, java.lang.String,
int, double, double, java.lang.String, java.lang.String, java.lang.String,
long, java.lang.String, java.lang.String, java.lang.String, java.lang.String,
long, long, long, long, java.lang.String, com.xyz.xyz.xyz.domain.Patient,
com.xyz.xyz.xyz.domain.Physician, scala.collection.Seq, scala.collection.Seq,
java.lang.String, long, java.lang.String, int, int, boolean, boolean,
scala.collection.Seq, boolean, scala.collection.Seq, boolean,
scala.collection.mutable.Seq, scala.collection.mutable.Seq)"
The relevant lines are:
build 17-Oct-2017 05:30:50 /* 093 */ private scala.collection.Seq
argValue84;
build 17-Oct-2017 05:30:50 /* 094 */ private scala.collection.Seq
argValue85;
and
build 17-Oct-2017 05:30:54 /* 8217 */ final
com.xyz.xyz.xyz.domain.Account value1 = false ? null : new
com.xyz.xyz.xyz.domain.Account(argValue2, argValue3, argValue4, argValue5,
argValue6, argValue7, argValue8, argValue9, argValue10, argValue11, argValue12,
argValue13, argValue14, argValue15, argValue16, argValue17, argValue18,
argValue19, argValue20, argValue21, argValue22, argValue23, argValue24,
argValue25, argValue26, argValue27, argValue28, argValue29, argValue30,
argValue31, argValue32, argValue33, argValue34, argValue35, argValue36,
argValue37, argValue38, argValue39, argValue40, argValue41, argValue42,
argValue43, argValue44, argValue45, argValue46, argValue47, argValue48,
argValue49, argValue50, argValue51, argValue52, argValue53, argValue54,
argValue55, argValue56, argValue57, argValue58, argValue59, argValue60,
argValue61, argValue62, argValue63, argValue64, argValue65, argValue66,
argValue67, argValue68, argValue69, argValue70, argValue71, argValue72,
argValue73, argValue74, argValue75, argValue76, argValue77, argValue78,
argValue79, argValue80, argValue81, argValue82, argValue83, argValue84,
argValue85);
In short, Spark uses scala.collection.Seq in the generated code which is not
compatible with scala.collection.mutable.Seq in our case class, which results
in a failure at runtime.
was:
We intermittently get this error when running Spark jobs. It is very
intermittent, I would estimate one in every 50-100 runs. We can't always
capture a full log file and our code is complex and 1000's of lines and the log
file is 533MB so I can't post it all. When it occurs we just run the exact
same job again and it runs fine, we do not have a repeatable case. I believe
it only happens in client mode, and not cluster mode.
Sorry I know this isn't a lot to go on but I wanted to report it since I see
other similar bugs like SPARK-19984 and SPARK-17936. If there is any other way
to collect more info or diagnostics please let me know.
build 17-Oct-2017 05:30:50 2017-10-17 09:30:50 [Executor task launch
worker-1] ERROR o.a.s.s.c.e.codegen.CodeGenerator - failed to compile:
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line
8217, Column 70: No applicable constructor/method found for actual parameters
"java.lang.String, java.lang.String, long, java.lang.String, long, long, long,
java.lang.String, long, long, double, scala.Option, scala.Option,
java.lang.String, java.lang.String, long, java.lang.String, java.lang.String,
java.lang.String, java.lang.String, java.lang.String, java.lang.String, int,
java.lang.String, java.lang.String, java.lang.String, java.lang.String,
java.lang.String, java.lang.String, java.lang.String, java.lang.String,
java.lang.String, java.lang.String, long, long, long, long, long, scala.Option,
scala.Option, scala.Option, scala.Option, scala.Option, java.lang.String,
java.lang.String, java.lang.String, java.lang.String, java.lang.String,
java.lang.String, long, java.lang.String, int, double, double,
java.lang.String, java.lang.String, java.lang.String, long, java.lang.String,
java.lang.String, java.lang.String, java.lang.String, long, long, long, long,
java.lang.String, com.xyz.xyz.xyz.domain.Patient,
com.xyz.xyz.xyz.domain.Physician, scala.collection.Seq, scala.collection.Seq,
java.lang.String, long, java.lang.String, int, int, boolean, boolean,
scala.collection.Seq, boolean, scala.collection.Seq, boolean,
scala.collection.Seq, scala.collection.Seq"; candidates are:
"com.xyz.xyz.xyz.domain.Account(java.lang.String, java.lang.String, long,
java.lang.String, long, long, long, java.lang.String, long, long, double,
scala.Option, scala.Option, java.lang.String, java.lang.String, long,
java.lang.String, java.lang.String, java.lang.String, java.lang.String,
java.lang.String, java.lang.String, int, java.lang.String, java.lang.String,
java.lang.String, java.lang.String, java.lang.String, java.lang.String,
java.lang.String, java.lang.String, java.lang.String, java.lang.String, long,
long, long, long, long, scala.Option, scala.Option, scala.Option, scala.Option,
scala.Option, java.lang.String, java.lang.String, java.lang.String,
java.lang.String, java.lang.String, java.lang.String, long, java.lang.String,
int, double, double, java.lang.String, java.lang.String, java.lang.String,
long, java.lang.String, java.lang.String, java.lang.String, java.lang.String,
long, long, long, long, java.lang.String, com.xyz.xyz.xyz.domain.Patient,
com.xyz.xyz.xyz.domain.Physician, scala.collection.Seq, scala.collection.Seq,
java.lang.String, long, java.lang.String, int, int, boolean, boolean,
scala.collection.Seq, boolean, scala.collection.Seq, boolean,
scala.collection.mutable.Seq, scala.collection.mutable.Seq)"
The relevant line is:
build 17-Oct-2017 05:30:54 /* 8217 */ final
com.xyz.xyz.xyz.domain.Account value1 = false ? null : new
com.xyz.xyz.xyz.domain.Account(argValue2, argValue3, argValue4, argValue5,
argValue6, argValue7, argValue8, argValue9, argValue10, argValue11, argValue12,
argValue13, argValue14, argValue15, argValue16, argValue17, argValue18,
argValue19, argValue20, argValue21, argValue22, argValue23, argValue24,
argValue25, argValue26, argValue27, argValue28, argValue29, argValue30,
argValue31, argValue32, argValue33, argValue34, argValue35, argValue36,
argValue37, argValue38, argValue39, argValue40, argValue41, argValue42,
argValue43, argValue44, argValue45, argValue46, argValue47, argValue48,
argValue49, argValue50, argValue51, argValue52, argValue53, argValue54,
argValue55, argValue56, argValue57, argValue58, argValue59, argValue60,
argValue61, argValue62, argValue63, argValue64, argValue65, argValue66,
argValue67, argValue68, argValue69, argValue70, argValue71, argValue72,
argValue73, argValue74, argValue75, argValue76, argValue77, argValue78,
argValue79, argValue80, argValue81, argValue82, argValue83, argValue84,
argValue85);
> CodeGenerator - failed to compile:
> org.codehaus.commons.compiler.CompileException: File 'generated.java'
> --------------------------------------------------------------------------------------------------------
>
> Key: SPARK-22296
> URL: https://issues.apache.org/jira/browse/SPARK-22296
> Project: Spark
> Issue Type: Bug
> Components: Optimizer
> Affects Versions: 2.1.0
> Reporter: Randy Tidd
>
> This is with Scala 2.11.
> We have a case class that has a constructor with 85 args, the last two of
> which are:
> var chargesInst :
> scala.collection.mutable.Seq[ChargeInstitutional] =
> scala.collection.mutable.Seq.empty[ChargeInstitutional],
> var chargesProf :
> scala.collection.mutable.Seq[ChargeProfessional] =
> scala.collection.mutable.Seq.empty[ChargeProfessional]
> A mutable Seq in a the constructor of a case class is probably poor form but
> Scala allows it. When we run this job we get this error:
> build 17-Oct-2017 05:30:50 2017-10-17 09:30:50 [Executor task launch
> worker-1] ERROR o.a.s.s.c.e.codegen.CodeGenerator - failed to compile:
> org.codehaus.commons.compiler.CompileException: File 'generated.java', Line
> 8217, Column 70: No applicable constructor/method found for actual parameters
> "java.lang.String, java.lang.String, long, java.lang.String, long, long,
> long, java.lang.String, long, long, double, scala.Option, scala.Option,
> java.lang.String, java.lang.String, long, java.lang.String, java.lang.String,
> java.lang.String, java.lang.String, java.lang.String, java.lang.String, int,
> java.lang.String, java.lang.String, java.lang.String, java.lang.String,
> java.lang.String, java.lang.String, java.lang.String, java.lang.String,
> java.lang.String, java.lang.String, long, long, long, long, long,
> scala.Option, scala.Option, scala.Option, scala.Option, scala.Option,
> java.lang.String, java.lang.String, java.lang.String, java.lang.String,
> java.lang.String, java.lang.String, long, java.lang.String, int, double,
> double, java.lang.String, java.lang.String, java.lang.String, long,
> java.lang.String, java.lang.String, java.lang.String, java.lang.String, long,
> long, long, long, java.lang.String, com.xyz.xyz.xyz.domain.Patient,
> com.xyz.xyz.xyz.domain.Physician, scala.collection.Seq, scala.collection.Seq,
> java.lang.String, long, java.lang.String, int, int, boolean, boolean,
> scala.collection.Seq, boolean, scala.collection.Seq, boolean,
> scala.collection.Seq, scala.collection.Seq"; candidates are:
> "com.xyz.xyz.xyz.domain.Account(java.lang.String, java.lang.String, long,
> java.lang.String, long, long, long, java.lang.String, long, long, double,
> scala.Option, scala.Option, java.lang.String, java.lang.String, long,
> java.lang.String, java.lang.String, java.lang.String, java.lang.String,
> java.lang.String, java.lang.String, int, java.lang.String, java.lang.String,
> java.lang.String, java.lang.String, java.lang.String, java.lang.String,
> java.lang.String, java.lang.String, java.lang.String, java.lang.String, long,
> long, long, long, long, scala.Option, scala.Option, scala.Option,
> scala.Option, scala.Option, java.lang.String, java.lang.String,
> java.lang.String, java.lang.String, java.lang.String, java.lang.String, long,
> java.lang.String, int, double, double, java.lang.String, java.lang.String,
> java.lang.String, long, java.lang.String, java.lang.String, java.lang.String,
> java.lang.String, long, long, long, long, java.lang.String,
> com.xyz.xyz.xyz.domain.Patient, com.xyz.xyz.xyz.domain.Physician,
> scala.collection.Seq, scala.collection.Seq, java.lang.String, long,
> java.lang.String, int, int, boolean, boolean, scala.collection.Seq, boolean,
> scala.collection.Seq, boolean, scala.collection.mutable.Seq,
> scala.collection.mutable.Seq)"
> The relevant lines are:
> build 17-Oct-2017 05:30:50 /* 093 */ private scala.collection.Seq
> argValue84;
> build 17-Oct-2017 05:30:50 /* 094 */ private scala.collection.Seq
> argValue85;
> and
> build 17-Oct-2017 05:30:54 /* 8217 */ final
> com.xyz.xyz.xyz.domain.Account value1 = false ? null : new
> com.xyz.xyz.xyz.domain.Account(argValue2, argValue3, argValue4, argValue5,
> argValue6, argValue7, argValue8, argValue9, argValue10, argValue11,
> argValue12, argValue13, argValue14, argValue15, argValue16, argValue17,
> argValue18, argValue19, argValue20, argValue21, argValue22, argValue23,
> argValue24, argValue25, argValue26, argValue27, argValue28, argValue29,
> argValue30, argValue31, argValue32, argValue33, argValue34, argValue35,
> argValue36, argValue37, argValue38, argValue39, argValue40, argValue41,
> argValue42, argValue43, argValue44, argValue45, argValue46, argValue47,
> argValue48, argValue49, argValue50, argValue51, argValue52, argValue53,
> argValue54, argValue55, argValue56, argValue57, argValue58, argValue59,
> argValue60, argValue61, argValue62, argValue63, argValue64, argValue65,
> argValue66, argValue67, argValue68, argValue69, argValue70, argValue71,
> argValue72, argValue73, argValue74, argValue75, argValue76, argValue77,
> argValue78, argValue79, argValue80, argValue81, argValue82, argValue83,
> argValue84, argValue85);
> In short, Spark uses scala.collection.Seq in the generated code which is not
> compatible with scala.collection.mutable.Seq in our case class, which results
> in a failure at runtime.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
