Github user myui commented on a diff in the pull request:
https://github.com/apache/incubator-hivemall/pull/111#discussion_r136031740
--- Diff: core/src/main/java/hivemall/recommend/SlimUDTF.java ---
@@ -77,6 +106,26 @@ public StructObjectInspector
initialize(ObjectInspector[] argOIs) throws UDFArgu
List<String> fieldNames = new ArrayList<>();
List<ObjectInspector> fieldOIs = new ArrayList<>();
+ // initialize temporary file to save knn for iterative training
+ if (mapredContext != null && numIterations > 1) {
+ // invoke only at task node (initialize is also invoked in
compilation)
+ final File file;
+ try {
+ file = File.createTempFile("hivemall_slim", ".sgmt"); //
A, Knn and R
+ file.deleteOnExit();
+ if (!file.canWrite()) {
+ throw new UDFArgumentException("Cannot write a
temporary file: "
+ + file.getAbsolutePath());
+ }
+ } catch (IOException ioe) {
+ throw new UDFArgumentException(ioe);
+ } catch (Throwable e) {
+ throw new UDFArgumentException(e);
+ }
+ this.fileIO = new NioStatefullSegment(file,false);
+ this.inputBuf = ByteBuffer.allocateDirect(1024*1024); // 1MB
--- End diff --
1MB might be too small to store KNNi. 4~8MB is enough (?). Estimate it by
recordBytes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
