Re: Review Request 65480: HIVE-18545 Add UDF to parse complex types from json

2018-06-19 Thread Zoltan Haindrich

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65480/
---

(Updated June 19, 2018, 7:02 a.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

update patch6; does not moves serde2.jsonserde


Bugs: HIVE-18545
https://issues.apache.org/jira/browse/HIVE-18545


Repository: hive-git


Description (updated)
---

* add a generalized reader for json2hive struct parsing
* add json_read udf ; tests
* make the hcat jsonserde use the "serde2" version as a back end - so that only 
1 real jsonserde exists in hive right now...


Diffs (updated)
-

  hcatalog/core/src/main/java/org/apache/hive/hcatalog/data/JsonSerDe.java 
114c205c83 
  hcatalog/core/src/test/java/org/apache/hive/hcatalog/data/TestJsonSerDe.java 
8aeb4f4f10 
  
itests/hive-jmh/src/main/java/org/apache/hive/benchmark/udf/json_read/JsonReadBench.java
 PRE-CREATION 
  
itests/hive-jmh/src/main/resources/org/apache/hive/benchmark/udf/json_read/val1.json
 PRE-CREATION 
  
itests/hive-jmh/src/main/resources/org/apache/hive/benchmark/udf/json_read/val1.type
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java e77fe18129 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFJsonRead.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFJsonRead.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/json_serde2.q PRE-CREATION 
  ql/src/test/queries/clientpositive/udf_json_read.q PRE-CREATION 
  ql/src/test/results/clientpositive/json_serde2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/show_functions.q.out 91d3660aa0 
  ql/src/test/results/clientpositive/udf_json_read.q.out PRE-CREATION 
  serde/src/java/org/apache/hadoop/hive/serde2/JsonSerDe.java 1119fa2af5 
  serde/src/java/org/apache/hadoop/hive/serde2/json/HiveJsonStructReader.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java cabb64cdcc 


Diff: https://reviews.apache.org/r/65480/diff/3/

Changes: https://reviews.apache.org/r/65480/diff/2-3/


Testing
---

* added some new tests for the udf


Thanks,

Zoltan Haindrich



Re: Review Request 65480: HIVE-18545 Add UDF to parse complex types from json

2018-06-15 Thread Zoltan Haindrich

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/65480/
---

(Updated June 15, 2018, 2:12 p.m.)


Review request for hive and Ashutosh Chauhan.


Changes
---

use the same in serde/udf; some refactor


Bugs: HIVE-18545
https://issues.apache.org/jira/browse/HIVE-18545


Repository: hive-git


Description (updated)
---

* add a generalized reader for json2hive struct parsing
* add json_read udf ; tests
* rename the new "copied" the new jsonserde "serde2.JsonSerde" to 
"serde2.json.JsonSerde2" - I think it would be bad to have 2 classes named the 
same ; in the same project, it would even confuse users - because only the 
package path was different
* make the hcat jsonserde use the "serde2" version as a back end - so that only 
1 real jsonserde exists in hive right now...


Diffs (updated)
-

  hcatalog/core/src/main/java/org/apache/hive/hcatalog/data/JsonSerDe.java 
114c205c83 
  hcatalog/core/src/test/java/org/apache/hive/hcatalog/data/TestJsonSerDe.java 
8aeb4f4f10 
  
itests/hive-jmh/src/main/java/org/apache/hive/benchmark/udf/json_read/JsonReadBench.java
 PRE-CREATION 
  
itests/hive-jmh/src/main/resources/org/apache/hive/benchmark/udf/json_read/val1.json
 PRE-CREATION 
  
itests/hive-jmh/src/main/resources/org/apache/hive/benchmark/udf/json_read/val1.type
 PRE-CREATION 
  ql/src/java/org/apache/hadoop/hive/ql/exec/FunctionRegistry.java e77fe18129 
  ql/src/java/org/apache/hadoop/hive/ql/udf/generic/GenericUDFJsonRead.java 
PRE-CREATION 
  ql/src/test/org/apache/hadoop/hive/ql/udf/generic/TestGenericUDFJsonRead.java 
PRE-CREATION 
  ql/src/test/queries/clientpositive/json_serde2.q PRE-CREATION 
  ql/src/test/queries/clientpositive/udf_json_read.q PRE-CREATION 
  ql/src/test/results/clientpositive/json_serde2.q.out PRE-CREATION 
  ql/src/test/results/clientpositive/show_functions.q.out 91d3660aa0 
  ql/src/test/results/clientpositive/udf_json_read.q.out PRE-CREATION 
  serde/src/java/org/apache/hadoop/hive/serde2/JsonSerDe.java 1119fa2af5 
  serde/src/java/org/apache/hadoop/hive/serde2/json/HiveJsonStructReader.java 
PRE-CREATION 
  serde/src/java/org/apache/hadoop/hive/serde2/json/JsonSerDe2.java 
PRE-CREATION 
  streaming/src/java/org/apache/hive/streaming/StrictJsonWriter.java cabb64cdcc 


Diff: https://reviews.apache.org/r/65480/diff/2/

Changes: https://reviews.apache.org/r/65480/diff/1-2/


Testing (updated)
---

* added some new tests for the udf


Thanks,

Zoltan Haindrich