Github user yanboliang commented on a diff in the pull request:
https://github.com/apache/spark/pull/14519#discussion_r73881801
--- Diff:
mllib/src/main/scala/org/apache/spark/ml/regression/AFTSurvivalRegression.scala
---
@@ -478,21 +482,23 @@ object AFTSurvivalRegressionModel extends
MLReadable[AFTSurvivalRegressionModel]
* $$
* </blockquote></p>
*
- * @param parameters including three part: The log of scale parameter, the
intercept and
- * regression coefficients corresponding to the features.
+ * @param bcParameters The broadcasted value includes three part: The log
of scale parameter,
+ * the intercept and regression coefficients
corresponding to the features.
* @param fitIntercept Whether to fit an intercept term.
- * @param featuresStd The standard deviation values of the features.
+ * @param bcFeaturesStd The broadcast standard deviation values of the
features.
*/
private class AFTAggregator(
- parameters: BDV[Double],
+ bcParameters: Broadcast[BDV[Double]],
fitIntercept: Boolean,
- featuresStd: Array[Double]) extends Serializable {
+ bcFeaturesStd: Broadcast[Array[Double]]) extends Serializable {
+ // make transient so we do not serialize between aggregation stages
+ @transient private lazy val parameters = bcParameters.value
// the regression coefficients to the covariates
- private val coefficients = parameters.slice(2, parameters.length)
- private val intercept = parameters(1)
+ @transient private lazy val coefficients = parameters.slice(2,
parameters.length)
+ @transient private lazy val intercept = parameters(1)
// sigma is the scale parameter of the AFT model
- private val sigma = math.exp(parameters(0))
+ @transient private lazy val sigma = math.exp(parameters(0))
--- End diff --
@dbtsai I addressed the ```parameters.length``` issue. But I can not remove
```lazy``` from ```@transient private lazy val parameters =
bcParameters.value``` and ```intercept/sigma```. Otherwise, it complains
```NullPointerException```. If I removed both ```@transient``` and ```lazy```,
it works well, but this does not coincide with our requirements. It's a little
weird and I'm still work on to figure out the root cause, can you give me some
suggestion? Thanks.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
