Dear Wiki user, You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.
The "SerDe" page has been changed by PradeepKamath. http://wiki.apache.org/hadoop/SerDe?action=diff&rev1=9&rev2=10 -------------------------------------------------- Output is analogous to input. The engine passes the deserialized Object representing a record and the corresponding !ObjectInspector to Serde.serialize(). In this context serialization means converting the record object to an object of the type expected by the !OutputFormat which will be used to perform the write. To perform this conversion, the serialize() method can make use of the passed !ObjectInspector to get the individual fields in the record in order to convert the record to the appropriate type. = Additional notes = - 1. The owner of an object (either a row, a column, a sub field of a column, or the return value of a UDF) is the code that creates it, and the life time of an object expires when the corresponding object for the next row is created. That means several things: + * The owner of an object (either a row, a column, a sub field of a column, or the return value of a UDF) is the code that creates it, and the life time of an object expires when the corresponding object for the next row is created. That means several things: - 1.a. We should not directly cache any object. In both group by and join, we copy the object and then put into a hashmap. + * We should not directly cache any object. In both group by and join, we copy the object and then put into a hashmap. - 1.b. SerDe, UDF, etc can reuse the same object for the same column in different rows. That means we can get rid of most of the object creations in the data pipeline, which is a huge performance boost. + * SerDe, UDF, etc can reuse the same object for the same column in different rows. That means we can get rid of most of the object creations in the data pipeline, which is a huge performance boost. - 2. Settable !ObjectInspectors (for write and object creation). + * Settable !ObjectInspectors (for write and object creation). - 2.a. !ObjectInspector allows us to "get" fields, but not "set" fields or "create" objects. + * !ObjectInspector allows us to "get" fields, but not "set" fields or "create" objects. - 2.b. Settable !ObjectInspectors allows that. + * Settable !ObjectInspectors allows that. - 2.c. We can convert an object with !JavaIntObjectInspector to an object with !WritableIntObjectInspector (which is, from Integer to !IntWritable) easily with the help of Settable !ObjectInspectors. + * We can convert an object with !JavaIntObjectInspector to an object with !WritableIntObjectInspector (which is, from Integer to !IntWritable) easily with the help of Settable !ObjectInspectors. - 2.d. In UDFs (non-GenericUDFs), we use Java reflection to get the type of the parameters/return values of a function (like !IntWritable in case of UDFOPPlus), and then infer the !ObjectInspector for that using !ObjectInspectorUtils.getStandardObjectInspectors. + * In UDFs (non-GenericUDFs), we use Java reflection to get the type of the parameters/return values of a function (like !IntWritable in case of UDFOPPlus), and then infer the !ObjectInspector for that using !ObjectInspectorUtils.getStandardObjectInspectors. - 2.e. Given the !ObjectInspector of an Object that is passed to a UDF, and the !ObjectInspector of the type of the parameter of the UDF, we will construct a !ObjectInspectorConverter, which uses the !SettableObjectInspector interface to convert the object. The converters are called in GenericUDF and GenericUDAF. + * Given the !ObjectInspector of an Object that is passed to a UDF, and the !ObjectInspector of the type of the parameter of the UDF, we will construct a !ObjectInspectorConverter, which uses the !SettableObjectInspector interface to convert the object. The converters are called in GenericUDF and GenericUDAF. In short, Hive will automatically convert objects so that Integer will be converted to !IntWritable (and vice versa) if needed. This allows people without Hadoop knowledge to use Java primitive classes (Integer, etc), while hadoop users/experts can use !IntWritable which is more efficient.
