We can reduce the default one, but usually a will user increase/change that when tuning hyper-parameters to increase the accuracy. So we need a solution that would work globally (for any value). A typical 'user'' cannot/shouldn't enable checkpointing, as IMO its a server configuration.
anyway, the default one is 20, which is still in the lower side :) On Wed, Aug 31, 2016 at 7:15 PM, Nirmal Fernando <[email protected]> wrote: > Can't we reduce the default number of iterations? and document how to > enable 'check pointing'. > > On Wed, Aug 31, 2016 at 7:03 PM, Supun Sethunga <[email protected]> wrote: > >> Hi all, >> >> We are encountering $subject in ML, for the default hyper-parameter >> values. A similar issue has been reported in [1], but with a different >> algorithm. >> >> This occurs when the number of iterations for model training is large. >> The solution suggested at [1] (setting a checkpoint directory) works for >> our scenario, and is the only solid solution we have for the moment. But as >> mentioned in [2], checkpointing add some overhead for spark operations, and >> requires some tuning based on the use case. Therefore, I'm not sure is it a >> good idea to enable checkpointing in ML, as it would affect DAS's >> performance. (This checkpointing is done for the Spark Context, and it is >> shared by both ML and DAS) >> >> Other option would be to, set checkpointing at the start of the >> Recommendation algorithm, and once the model is trained, then unset >> checkpointing. Since we are encountering this issue only at this particular >> algorithm, it is not needed to be done for any other algorithm. >> >> Would like to know what would be the best approach? >> >> [1] https://issues.apache.org/jira/browse/SPARK-13546 >> [2] http://spark.apache.org/docs/1.6.2/streaming-programming >> -guide.html#checkpointing >> >> >> >> *Stack Trace:* >> >> Caused by: java.lang.StackOverflowError >> at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Obje >> ctInputStream.java:2606) >> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1506) >> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >> am.java:1774) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >> m.java:2000) >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >> am.java:1801) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >> m.java:2000) >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >> am.java:1801) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >> m.java:2000) >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >> am.java:1801) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) >> at scala.collection.immutable.$colon$colon.readObject(List.scala:362) >> at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source) >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >> thodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:497) >> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass >> .java:1058) >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900) >> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >> am.java:1801) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >> m.java:2000) >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >> am.java:1801) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >> m.java:2000) >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >> am.java:1801) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) >> at scala.collection.immutable.$colon$colon.readObject(List.scala:366) >> at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source) >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >> thodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:497) >> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass >> .java:1058) >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900) >> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >> am.java:1801) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >> m.java:2000) >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >> am.java:1801) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >> m.java:2000) >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >> am.java:1801) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) >> at scala.collection.immutable.$colon$colon.readObject(List.scala:362) >> at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source) >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >> thodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:497) >> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass >> .java:1058) >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900) >> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >> am.java:1801) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >> m.java:2000) >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >> am.java:1801) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >> m.java:2000) >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >> am.java:1801) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) >> at scala.collection.immutable.$colon$colon.readObject(List.scala:362) >> at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source) >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >> thodAccessorImpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:497) >> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass >> .java:1058) >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900) >> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >> am.java:1801) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >> m.java:2000) >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >> am.java:1801) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >> m.java:2000) >> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >> am.java:1801) >> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) >> at scala.collection.immutable.$colon$colon.readObject(List.scala:362) >> at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source) >> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >> thodAccessorImpl.java:43) >> >> >> Regards, >> Supun >> -- >> *Supun Sethunga* >> Senior Software Engineer >> WSO2, Inc. >> http://wso2.com/ >> lean | enterprise | middleware >> Mobile : +94 716546324 >> Blog: http://supunsetunga.blogspot.com >> > > > > -- > > Thanks & regards, > Nirmal > > Team Lead - WSO2 Machine Learner > Associate Technical Lead - Data Technologies Team, WSO2 Inc. > Mobile: +94715779733 > Blog: http://nirmalfdo.blogspot.com/ > > > -- *Supun Sethunga* Senior Software Engineer WSO2, Inc. http://wso2.com/ lean | enterprise | middleware Mobile : +94 716546324 Blog: http://supunsetunga.blogspot.com
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
