Hi,
I integrated the apache spark decision tree classifier in a java
program that reads real time data into an array called 'vals' and then
run the code:
Vector v = Vectors.dense(vals);
LabeledPoint pos = new LabeledPoint(0.0, v);
SparkConf sparkConf = new
SparkConf().setAppName("ContactListenerExample").setMaster("local[2]").set("spark.executor.memory","1g");
SparkContext sc = new SparkContext(sparkConf);
DecisionTreeModel dmodel = DecisionTreeModel.load(sc,
"C:/Users/super/eclipse-workspace/javaspark/tdata/tennisdecmodel.model");
double pred = dmodel.predict(pos.features());
System.out.println("prediction is:"+pred);
... But it does not print out the prediction and instead prints out
the following:
(Do you know why it is printing this out?)
[DEBUG] 2019-05-02 18:19:49.875 [Executor task launch worker for task
2] ParquetMetadataConverter - {
"fileMetaData" : {
"schema" : {
"name" : "spark_schema",
"repetition" : "REPEATED",
"originalType" : null,
"id" : null,
"fields" : [ {
"name" : "treeId",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "INT32",
"decimalMetadata" : null,
"typeLength" : 0
}, {
"name" : "nodeId",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "INT32",
"decimalMetadata" : null,
"typeLength" : 0
}, {
"name" : "predict",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"fields" : [ {
"name" : "predict",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "DOUBLE",
"decimalMetadata" : null,
"typeLength" : 0
}, {
"name" : "prob",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "DOUBLE",
"decimalMetadata" : null,
"typeLength" : 0
} ],
"primitive" : false,
"fieldCount" : 2
}, {
"name" : "impurity",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "DOUBLE",
"decimalMetadata" : null,
"typeLength" : 0
}, {
"name" : "isLeaf",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "BOOLEAN",
"decimalMetadata" : null,
"typeLength" : 0
}, {
"name" : "split",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"fields" : [ {
"name" : "feature",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "INT32",
"decimalMetadata" : null,
"typeLength" : 0
}, {
"name" : "threshold",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "DOUBLE",
"decimalMetadata" : null,
"typeLength" : 0
}, {
"name" : "featureType",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "INT32",
"decimalMetadata" : null,
"typeLength" : 0
}, {
"name" : "categories",
"repetition" : "OPTIONAL",
"originalType" : "LIST",
"id" : null,
"fields" : [ {
"name" : "list",
"repetition" : "REPEATED",
"originalType" : null,
"id" : null,
"fields" : [ {
"name" : "element",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "DOUBLE",
"decimalMetadata" : null,
"typeLength" : 0
} ],
"primitive" : false,
"fieldCount" : 1
} ],
"primitive" : false,
"fieldCount" : 1
} ],
"primitive" : false,
"fieldCount" : 4
}, {
"name" : "leftNodeId",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "INT32",
"decimalMetadata" : null,
"typeLength" : 0
}, {
"name" : "rightNodeId",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "INT32",
"decimalMetadata" : null,
"typeLength" : 0
}, {
"name" : "infoGain",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "DOUBLE",
"decimalMetadata" : null,
"typeLength" : 0
} ],
"columns" : [ {
"path" : [ "treeId" ],
"type" : "INT32",
"primitiveType" : {
"name" : "treeId",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "INT32",
"decimalMetadata" : null,
"typeLength" : 0
},
"maxDefinitionLevel" : 0,
"maxRepetitionLevel" : 0,
"typeLength" : 0
}, {
"path" : [ "nodeId" ],
"type" : "INT32",
"primitiveType" : {
"name" : "nodeId",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "INT32",
"decimalMetadata" : null,
"typeLength" : 0
},
"maxDefinitionLevel" : 0,
"maxRepetitionLevel" : 0,
"typeLength" : 0
}, {
"path" : [ "predict", "predict" ],
"type" : "DOUBLE",
"primitiveType" : {
"name" : "predict",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "DOUBLE",
"decimalMetadata" : null,
"typeLength" : 0
},
"maxDefinitionLevel" : 1,
"maxRepetitionLevel" : 0,
"typeLength" : 0
}, {
"path" : [ "predict", "prob" ],
"type" : "DOUBLE",
"primitiveType" : {
"name" : "prob",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "DOUBLE",
"decimalMetadata" : null,
"typeLength" : 0
},
"maxDefinitionLevel" : 1,
"maxRepetitionLevel" : 0,
"typeLength" : 0
}, {
"path" : [ "impurity" ],
"type" : "DOUBLE",
"primitiveType" : {
"name" : "impurity",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "DOUBLE",
"decimalMetadata" : null,
"typeLength" : 0
},
"maxDefinitionLevel" : 0,
"maxRepetitionLevel" : 0,
"typeLength" : 0
}, {
"path" : [ "isLeaf" ],
"type" : "BOOLEAN",
"primitiveType" : {
"name" : "isLeaf",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "BOOLEAN",
"decimalMetadata" : null,
"typeLength" : 0
},
"maxDefinitionLevel" : 0,
"maxRepetitionLevel" : 0,
"typeLength" : 0
}, {
"path" : [ "split", "feature" ],
"type" : "INT32",
"primitiveType" : {
"name" : "feature",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "INT32",
"decimalMetadata" : null,
"typeLength" : 0
},
"maxDefinitionLevel" : 1,
"maxRepetitionLevel" : 0,
"typeLength" : 0
}, {
"path" : [ "split", "threshold" ],
"type" : "DOUBLE",
"primitiveType" : {
"name" : "threshold",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "DOUBLE",
"decimalMetadata" : null,
"typeLength" : 0
},
"maxDefinitionLevel" : 1,
"maxRepetitionLevel" : 0,
"typeLength" : 0
}, {
"path" : [ "split", "featureType" ],
"type" : "INT32",
"primitiveType" : {
"name" : "featureType",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "INT32",
"decimalMetadata" : null,
"typeLength" : 0
},
"maxDefinitionLevel" : 1,
"maxRepetitionLevel" : 0,
"typeLength" : 0
}, {
"path" : [ "split", "categories", "list", "element" ],
"type" : "DOUBLE",
"primitiveType" : {
"name" : "element",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "DOUBLE",
"decimalMetadata" : null,
"typeLength" : 0
},
"maxDefinitionLevel" : 3,
"maxRepetitionLevel" : 1,
"typeLength" : 0
}, {
"path" : [ "leftNodeId" ],
"type" : "INT32",
"primitiveType" : {
"name" : "leftNodeId",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "INT32",
"decimalMetadata" : null,
"typeLength" : 0
},
"maxDefinitionLevel" : 1,
"maxRepetitionLevel" : 0,
"typeLength" : 0
}, {
"path" : [ "rightNodeId" ],
"type" : "INT32",
"primitiveType" : {
"name" : "rightNodeId",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "INT32",
"decimalMetadata" : null,
"typeLength" : 0
},
"maxDefinitionLevel" : 1,
"maxRepetitionLevel" : 0,
"typeLength" : 0
}, {
"path" : [ "infoGain" ],
"type" : "DOUBLE",
"primitiveType" : {
"name" : "infoGain",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "DOUBLE",
"decimalMetadata" : null,
"typeLength" : 0
},
"maxDefinitionLevel" : 1,
"maxRepetitionLevel" : 0,
"typeLength" : 0
} ],
"paths" : [ [ "treeId" ], [ "nodeId" ], [ "predict", "predict"
], [ "predict", "prob" ], [ "impurity" ], [ "isLeaf" ], [ "split",
"feature" ], [ "split", "threshold" ], [ "split", "featureType" ], [
"split", "categories", "list", "element" ], [ "leftNodeId" ], [
"rightNodeId" ], [ "infoGain" ] ],
"primitive" : false,
"fieldCount" : 9
},
"keyValueMetaData" : {
"org.apache.spark.sql.parquet.row.metadata" :
"{\"type\":\"struct\",\"fields\":[{\"name\":\"treeId\",\"type\":\"integer\",\"nullable\":false,\"metadata\":{}},{\"name\":\"nodeId\",\"type\":\"integer\",\"nullable\":false,\"metadata\":{}},{\"name\":\"predict\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"predict\",\"type\":\"double\",\"nullable\":false,\"metadata\":{}},{\"name\":\"prob\",\"type\":\"double\",\"nullable\":false,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}},{\"name\":\"impurity\",\"type\":\"double\",\"nullable\":false,\"metadata\":{}},{\"name\":\"isLeaf\",\"type\":\"boolean\",\"nullable\":false,\"metadata\":{}},{\"name\":\"split\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"feature\",\"type\":\"integer\",\"nullable\":false,\"metadata\":{}},{\"name\":\"threshold\",\"type\":\"double\",\"nullable\":false,\"metadata\":{}},{\"name\":\"featureType\",\"type\":\"integer\",\"nullable\":false,\"metadata\":{}},{\"name\":\"categories\",\"type\":{\"type\":\"array\",\"elementType\":\"double\",\"containsNull\":false},\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}},{\"name\":\"leftNodeId\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"rightNodeId\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"infoGain\",\"type\":\"double\",\"nullable\":true,\"metadata\":{}}]}"
},
"createdBy" : "parquet-mr (build 32c46643845ea8a705c35d4ec8fc654cc8ff816d)"
},
"blocks" : [ ]
}
[DEBUG] 2019-05-02 18:19:49.880 [Executor task launch worker for task
3] ParquetFileReader - File length 2716
[DEBUG] 2019-05-02 18:19:49.881 [Executor task launch worker for task
3] ParquetFileReader - reading footer index at 2708
[INFO ] 2019-05-02 18:19:49.882 [Thread-9] KRServer - INFO -- received
UDP frame data on port[9000], #bytes[440]
[INFO ] 2019-05-02 18:19:49.882 [Thread-9] KRServer -
SSCRTDataFrame(packetLen[436], packetType[0xa3], seqNo[4203],
frameType[13], frameTime[42009], sensorBits[B111101111011],
resentBits[B0])
[DEBUG] 2019-05-02 18:19:49.883 [Executor task launch worker for task
3] ParquetFileReader - read footer length: 2158, footer index: 550
[DEBUG] 2019-05-02 18:19:49.884 [Executor task launch worker for task
3] ParquetMetadataConverter - FileMetaData(version:1,
schema:[SchemaElement(name:spark_schema, num_children:9),
SchemaElement(type:INT32, repetition_type:REQUIRED, name:treeId),
SchemaElement(type:INT32, repetition_type:REQUIRED, name:nodeId),
SchemaElement(repetition_type:OPTIONAL, name:predict, num_children:2),
SchemaElement(type:DOUBLE, repetition_type:REQUIRED, name:predict),
SchemaElement(type:DOUBLE, repetition_type:REQUIRED, name:prob),
SchemaElement(type:DOUBLE, repetition_type:REQUIRED, name:impurity),
SchemaElement(type:BOOLEAN, repetition_type:REQUIRED, name:isLeaf),
SchemaElement(repetition_type:OPTIONAL, name:split, num_children:4),
SchemaElement(type:INT32, repetition_type:REQUIRED, name:feature),
SchemaElement(type:DOUBLE, repetition_type:REQUIRED, name:threshold),
SchemaElement(type:INT32, repetition_type:REQUIRED, name:featureType),
SchemaElement(repetition_type:OPTIONAL, name:categories,
num_children:1, converted_type:LIST),
SchemaElement(repetition_type:REPEATED, name:list, num_children:1),
SchemaElement(type:DOUBLE, repetition_type:REQUIRED, name:element),
SchemaElement(type:INT32, repetition_type:OPTIONAL, name:leftNodeId),
SchemaElement(type:INT32, repetition_type:OPTIONAL, name:rightNodeId),
SchemaElement(type:DOUBLE, repetition_type:OPTIONAL, name:infoGain)],
num_rows:2, row_groups:null,
key_value_metadata:[KeyValue(key:org.apache.spark.sql.parquet.row.metadata,
value:{"type":"struct","fields":[{"name":"treeId","type":"integer","nullable":false,"metadata":{}},{"name":"nodeId","type":"integer","nullable":false,"metadata":{}},{"name":"predict","type":{"type":"struct","fields":[{"name":"predict","type":"double","nullable":false,"metadata":{}},{"name":"prob","type":"double","nullable":false,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"impurity","type":"double","nullable":false,"metadata":{}},{"name":"isLeaf","type":"boolean","nullable":false,"metadata":{}},{"name":"split","type":{"type":"struct","fields":[{"name":"feature","type":"integer","nullable":false,"metadata":{}},{"name":"threshold","type":"double","nullable":false,"metadata":{}},{"name":"featureType","type":"integer","nullable":false,"metadata":{}},{"name":"categories","type":{"type":"array","elementType":"double","containsNull":false},"nullable":true,"metadata":{}}]},"nullable":true,"metadata":{}},{"name":"leftNodeId","type":"integer","nullable":true,"metadata":{}},{"name":"rightNodeId","type":"integer","nullable":true,"metadata":{}},{"name":"infoGain","type":"double","nullable":true,"metadata":{}}]})],
created_by:parquet-mr (build
32c46643845ea8a705c35d4ec8fc654cc8ff816d))
[DEBUG] 2019-05-02 18:19:49.888 [Executor task launch worker for task
3] ParquetMetadataConverter - {
"fileMetaData" : {
"schema" : {
"name" : "spark_schema",
"repetition" : "REPEATED",
"originalType" : null,
"id" : null,
"fields" : [ {
"name" : "treeId",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "INT32",
"decimalMetadata" : null,
"typeLength" : 0
}, {
"name" : "nodeId",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "INT32",
"decimalMetadata" : null,
"typeLength" : 0
}, {
"name" : "predict",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"fields" : [ {
"name" : "predict",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "DOUBLE",
"decimalMetadata" : null,
"typeLength" : 0
}, {
"name" : "prob",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "DOUBLE",
"decimalMetadata" : null,
"typeLength" : 0
} ],
"primitive" : false,
"fieldCount" : 2
}, {
"name" : "impurity",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "DOUBLE",
"decimalMetadata" : null,
"typeLength" : 0
}, {
"name" : "isLeaf",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "BOOLEAN",
"decimalMetadata" : null,
"typeLength" : 0
}, {
"name" : "split",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"fields" : [ {
"name" : "feature",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "INT32",
"decimalMetadata" : null,
"typeLength" : 0
}, {
"name" : "threshold",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "DOUBLE",
"decimalMetadata" : null,
"typeLength" : 0
}, {
"name" : "featureType",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "INT32",
"decimalMetadata" : null,
"typeLength" : 0
}, {
"name" : "categories",
"repetition" : "OPTIONAL",
"originalType" : "LIST",
"id" : null,
"fields" : [ {
"name" : "list",
"repetition" : "REPEATED",
"originalType" : null,
"id" : null,
"fields" : [ {
"name" : "element",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "DOUBLE",
"decimalMetadata" : null,
"typeLength" : 0
} ],
"primitive" : false,
"fieldCount" : 1
} ],
"primitive" : false,
"fieldCount" : 1
} ],
"primitive" : false,
"fieldCount" : 4
}, {
"name" : "leftNodeId",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "INT32",
"decimalMetadata" : null,
"typeLength" : 0
}, {
"name" : "rightNodeId",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "INT32",
"decimalMetadata" : null,
"typeLength" : 0
}, {
"name" : "infoGain",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "DOUBLE",
"decimalMetadata" : null,
"typeLength" : 0
} ],
"columns" : [ {
"path" : [ "treeId" ],
"type" : "INT32",
"primitiveType" : {
"name" : "treeId",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "INT32",
"decimalMetadata" : null,
"typeLength" : 0
},
"maxDefinitionLevel" : 0,
"maxRepetitionLevel" : 0,
"typeLength" : 0
}, {
"path" : [ "nodeId" ],
"type" : "INT32",
"primitiveType" : {
"name" : "nodeId",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "INT32",
"decimalMetadata" : null,
"typeLength" : 0
},
"maxDefinitionLevel" : 0,
"maxRepetitionLevel" : 0,
"typeLength" : 0
}, {
"path" : [ "predict", "predict" ],
"type" : "DOUBLE",
"primitiveType" : {
"name" : "predict",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "DOUBLE",
"decimalMetadata" : null,
"typeLength" : 0
},
"maxDefinitionLevel" : 1,
"maxRepetitionLevel" : 0,
"typeLength" : 0
}, {
"path" : [ "predict", "prob" ],
"type" : "DOUBLE",
"primitiveType" : {
"name" : "prob",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "DOUBLE",
"decimalMetadata" : null,
"typeLength" : 0
},
"maxDefinitionLevel" : 1,
"maxRepetitionLevel" : 0,
"typeLength" : 0
}, {
"path" : [ "impurity" ],
"type" : "DOUBLE",
"primitiveType" : {
"name" : "impurity",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "DOUBLE",
"decimalMetadata" : null,
"typeLength" : 0
},
"maxDefinitionLevel" : 0,
"maxRepetitionLevel" : 0,
"typeLength" : 0
}, {
"path" : [ "isLeaf" ],
"type" : "BOOLEAN",
"primitiveType" : {
"name" : "isLeaf",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "BOOLEAN",
"decimalMetadata" : null,
"typeLength" : 0
},
"maxDefinitionLevel" : 0,
"maxRepetitionLevel" : 0,
"typeLength" : 0
}, {
"path" : [ "split", "feature" ],
"type" : "INT32",
"primitiveType" : {
"name" : "feature",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "INT32",
"decimalMetadata" : null,
"typeLength" : 0
},
"maxDefinitionLevel" : 1,
"maxRepetitionLevel" : 0,
"typeLength" : 0
}, {
"path" : [ "split", "threshold" ],
"type" : "DOUBLE",
"primitiveType" : {
"name" : "threshold",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "DOUBLE",
"decimalMetadata" : null,
"typeLength" : 0
},
"maxDefinitionLevel" : 1,
"maxRepetitionLevel" : 0,
"typeLength" : 0
}, {
"path" : [ "split", "featureType" ],
"type" : "INT32",
"primitiveType" : {
"name" : "featureType",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "INT32",
"decimalMetadata" : null,
"typeLength" : 0
},
"maxDefinitionLevel" : 1,
"maxRepetitionLevel" : 0,
"typeLength" : 0
}, {
"path" : [ "split", "categories", "list", "element" ],
"type" : "DOUBLE",
"primitiveType" : {
"name" : "element",
"repetition" : "REQUIRED",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "DOUBLE",
"decimalMetadata" : null,
"typeLength" : 0
},
"maxDefinitionLevel" : 3,
"maxRepetitionLevel" : 1,
"typeLength" : 0
}, {
"path" : [ "leftNodeId" ],
"type" : "INT32",
"primitiveType" : {
"name" : "leftNodeId",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "INT32",
"decimalMetadata" : null,
"typeLength" : 0
},
"maxDefinitionLevel" : 1,
"maxRepetitionLevel" : 0,
"typeLength" : 0
}, {
"path" : [ "rightNodeId" ],
"type" : "INT32",
"primitiveType" : {
"name" : "rightNodeId",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "INT32",
"decimalMetadata" : null,
"typeLength" : 0
},
"maxDefinitionLevel" : 1,
"maxRepetitionLevel" : 0,
"typeLength" : 0
}, {
"path" : [ "infoGain" ],
"type" : "DOUBLE",
"primitiveType" : {
"name" : "infoGain",
"repetition" : "OPTIONAL",
"originalType" : null,
"id" : null,
"primitive" : true,
"primitiveTypeName" : "DOUBLE",
"decimalMetadata" : null,
"typeLength" : 0
},
"maxDefinitionLevel" : 1,
"maxRepetitionLevel" : 0,
"typeLength" : 0
} ],
"paths" : [ [ "treeId" ], [ "nodeId" ], [ "predict", "predict"
], [ "predict", "prob" ], [ "impurity" ], [ "isLeaf" ], [ "split",
"feature" ], [ "split", "threshold" ], [ "split", "featureType" ], [
"split", "categories", "list", "element" ], [ "leftNodeId" ], [
"rightNodeId" ], [ "infoGain" ] ],
"primitive" : false,
"fieldCount" : 9
},
"keyValueMetaData" : {
"org.apache.spark.sql.parquet.row.metadata" :
"{\"type\":\"struct\",\"fields\":[{\"name\":\"treeId\",\"type\":\"integer\",\"nullable\":false,\"metadata\":{}},{\"name\":\"nodeId\",\"type\":\"integer\",\"nullable\":false,\"metadata\":{}},{\"name\":\"predict\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"predict\",\"type\":\"double\",\"nullable\":false,\"metadata\":{}},{\"name\":\"prob\",\"type\":\"double\",\"nullable\":false,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}},{\"name\":\"impurity\",\"type\":\"double\",\"nullable\":false,\"metadata\":{}},{\"name\":\"isLeaf\",\"type\":\"boolean\",\"nullable\":false,\"metadata\":{}},{\"name\":\"split\",\"type\":{\"type\":\"struct\",\"fields\":[{\"name\":\"feature\",\"type\":\"integer\",\"nullable\":false,\"metadata\":{}},{\"name\":\"threshold\",\"type\":\"double\",\"nullable\":false,\"metadata\":{}},{\"name\":\"featureType\",\"type\":\"integer\",\"nullable\":false,\"metadata\":{}},{\"name\":\"categories\",\"type\":{\"type\":\"array\",\"elementType\":\"double\",\"containsNull\":false},\"nullable\":true,\"metadata\":{}}]},\"nullable\":true,\"metadata\":{}},{\"name\":\"leftNodeId\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"rightNodeId\",\"type\":\"integer\",\"nullable\":true,\"metadata\":{}},{\"name\":\"infoGain\",\"type\":\"double\",\"nullable\":true,\"metadata\":{}}]}"
},
"createdBy" : "parquet-mr (build 32c46643845ea8a705c35d4ec8fc654cc8ff816d)"
},
"blocks" : [ ]
}
--
Sian Ees Super.
---------------------------------------------------------------------
To unsubscribe e-mail: [email protected]
