'checkpointing' is an algorithm property right? We can add it as a hyperparameter configuration ? What do we specifically need to do as ML server?
On Wed, Aug 31, 2016 at 10:47 PM, Supun Sethunga <[email protected]> wrote: > We can reduce the default one, but usually a will user increase/change > that when tuning hyper-parameters to increase the accuracy. So we need a > solution that would work globally (for any value). A typical 'user'' > cannot/shouldn't enable checkpointing, as IMO its a server configuration. > > anyway, the default one is 20, which is still in the lower side :) > > On Wed, Aug 31, 2016 at 7:15 PM, Nirmal Fernando <[email protected]> wrote: > >> Can't we reduce the default number of iterations? and document how to >> enable 'check pointing'. >> >> On Wed, Aug 31, 2016 at 7:03 PM, Supun Sethunga <[email protected]> wrote: >> >>> Hi all, >>> >>> We are encountering $subject in ML, for the default hyper-parameter >>> values. A similar issue has been reported in [1], but with a different >>> algorithm. >>> >>> This occurs when the number of iterations for model training is large. >>> The solution suggested at [1] (setting a checkpoint directory) works for >>> our scenario, and is the only solid solution we have for the moment. But as >>> mentioned in [2], checkpointing add some overhead for spark operations, and >>> requires some tuning based on the use case. Therefore, I'm not sure is it a >>> good idea to enable checkpointing in ML, as it would affect DAS's >>> performance. (This checkpointing is done for the Spark Context, and it is >>> shared by both ML and DAS) >>> >>> Other option would be to, set checkpointing at the start of the >>> Recommendation algorithm, and once the model is trained, then unset >>> checkpointing. Since we are encountering this issue only at this particular >>> algorithm, it is not needed to be done for any other algorithm. >>> >>> Would like to know what would be the best approach? >>> >>> [1] https://issues.apache.org/jira/browse/SPARK-13546 >>> [2] http://spark.apache.org/docs/1.6.2/streaming-programming >>> -guide.html#checkpointing >>> >>> >>> >>> *Stack Trace:* >>> >>> Caused by: java.lang.StackOverflowError >>> at java.io.ObjectInputStream$BlockDataInputStream.peekByte(Obje >>> ctInputStream.java:2606) >>> at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1506) >>> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >>> am.java:1774) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >>> m.java:2000) >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >>> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >>> am.java:1801) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >>> m.java:2000) >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >>> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >>> am.java:1801) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >>> m.java:2000) >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >>> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >>> am.java:1801) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) >>> at scala.collection.immutable.$colon$colon.readObject(List.scala:362) >>> at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source) >>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >>> thodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:497) >>> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass >>> .java:1058) >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900) >>> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >>> am.java:1801) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >>> m.java:2000) >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >>> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >>> am.java:1801) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >>> m.java:2000) >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >>> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >>> am.java:1801) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) >>> at scala.collection.immutable.$colon$colon.readObject(List.scala:366) >>> at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source) >>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >>> thodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:497) >>> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass >>> .java:1058) >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900) >>> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >>> am.java:1801) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >>> m.java:2000) >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >>> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >>> am.java:1801) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >>> m.java:2000) >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >>> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >>> am.java:1801) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) >>> at scala.collection.immutable.$colon$colon.readObject(List.scala:362) >>> at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source) >>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >>> thodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:497) >>> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass >>> .java:1058) >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900) >>> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >>> am.java:1801) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >>> m.java:2000) >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >>> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >>> am.java:1801) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >>> m.java:2000) >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >>> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >>> am.java:1801) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) >>> at scala.collection.immutable.$colon$colon.readObject(List.scala:362) >>> at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source) >>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >>> thodAccessorImpl.java:43) >>> at java.lang.reflect.Method.invoke(Method.java:497) >>> at java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass >>> .java:1058) >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1900) >>> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >>> am.java:1801) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >>> m.java:2000) >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >>> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >>> am.java:1801) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> at java.io.ObjectInputStream.defaultReadFields(ObjectInputStrea >>> m.java:2000) >>> at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1924) >>> at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStre >>> am.java:1801) >>> at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1351) >>> at java.io.ObjectInputStream.readObject(ObjectInputStream.java:371) >>> at scala.collection.immutable.$colon$colon.readObject(List.scala:362) >>> at sun.reflect.GeneratedMethodAccessor51.invoke(Unknown Source) >>> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe >>> thodAccessorImpl.java:43) >>> >>> >>> Regards, >>> Supun >>> -- >>> *Supun Sethunga* >>> Senior Software Engineer >>> WSO2, Inc. >>> http://wso2.com/ >>> lean | enterprise | middleware >>> Mobile : +94 716546324 >>> Blog: http://supunsetunga.blogspot.com >>> >> >> >> >> -- >> >> Thanks & regards, >> Nirmal >> >> Team Lead - WSO2 Machine Learner >> Associate Technical Lead - Data Technologies Team, WSO2 Inc. >> Mobile: +94715779733 >> Blog: http://nirmalfdo.blogspot.com/ >> >> >> > > > -- > *Supun Sethunga* > Senior Software Engineer > WSO2, Inc. > http://wso2.com/ > lean | enterprise | middleware > Mobile : +94 716546324 > Blog: http://supunsetunga.blogspot.com > -- Thanks & regards, Nirmal Team Lead - WSO2 Machine Learner Associate Technical Lead - Data Technologies Team, WSO2 Inc. Mobile: +94715779733 Blog: http://nirmalfdo.blogspot.com/
_______________________________________________ Dev mailing list [email protected] http://wso2.org/cgi-bin/mailman/listinfo/dev
