[jira] [Updated] (HIVE-22360) MultiSerDe returns wrong results in last column when the loaded file has more columns than those in table schema
[ https://issues.apache.org/jira/browse/HIVE-22360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shubham Chaurasia updated HIVE-22360: - Attachment: HIVE-22360.3.patch > MultiSerDe returns wrong results in last column when the loaded file has more > columns than those in table schema > > > Key: HIVE-22360 > URL: https://issues.apache.org/jira/browse/HIVE-22360 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 4.0.0 >Reporter: Shubham Chaurasia >Assignee: Shubham Chaurasia >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22360.1.patch, HIVE-22360.2.patch, > HIVE-22360.3.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Repro steps: > Input file: > {code} > 1^,1^,^,0^,0^,0 > 2^,1^,^,0^,1^,0 > 3^,1^,^,0^,0^,0 > 4^,1^,^,0^,1^,0 > {code} > Queries: > {code} > CREATE TABLE n2(colA int, colB tinyint, colC timestamp, colD smallint, colE > smallint) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.MultiDelimitSerDe' > WITH SERDEPROPERTIES ("field.delim"="^,")STORED AS TEXTFILE; > LOAD DATA LOCAL INPATH '/Users/schaurasia/Documents/input_6_cols.csv' > OVERWRITE INTO TABLE n2; > select * from n2; > // wrong last column results here. > +--+--+--+--+--+ > | n2.cola | n2.colb | n2.colc | n2.cold | n2.cole | > +--+--+--+--+--+ > | 1| 1| NULL | 0| NULL | > | 2| 1| NULL | 0| NULL | > | 3| 1| NULL | 0| NULL | > | 4| 1| NULL | 0| NULL | > +--+--+--+--+--+ > {code} > Cause: > In multi-serde parsing, the total length calculation here: > https://github.com/apache/hive/blob/rel/release-3.1.2/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java#L308 > does not take extra fields into account. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22360) MultiSerDe returns wrong results in last column when the loaded file has more columns than those in table schema
[ https://issues.apache.org/jira/browse/HIVE-22360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shubham Chaurasia updated HIVE-22360: - Attachment: HIVE-22360.2.patch > MultiSerDe returns wrong results in last column when the loaded file has more > columns than those in table schema > > > Key: HIVE-22360 > URL: https://issues.apache.org/jira/browse/HIVE-22360 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 4.0.0 >Reporter: Shubham Chaurasia >Assignee: Shubham Chaurasia >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22360.1.patch, HIVE-22360.2.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Repro steps: > Input file: > {code} > 1^,1^,^,0^,0^,0 > 2^,1^,^,0^,1^,0 > 3^,1^,^,0^,0^,0 > 4^,1^,^,0^,1^,0 > {code} > Queries: > {code} > CREATE TABLE n2(colA int, colB tinyint, colC timestamp, colD smallint, colE > smallint) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.MultiDelimitSerDe' > WITH SERDEPROPERTIES ("field.delim"="^,")STORED AS TEXTFILE; > LOAD DATA LOCAL INPATH '/Users/schaurasia/Documents/input_6_cols.csv' > OVERWRITE INTO TABLE n2; > select * from n2; > // wrong last column results here. > +--+--+--+--+--+ > | n2.cola | n2.colb | n2.colc | n2.cold | n2.cole | > +--+--+--+--+--+ > | 1| 1| NULL | 0| NULL | > | 2| 1| NULL | 0| NULL | > | 3| 1| NULL | 0| NULL | > | 4| 1| NULL | 0| NULL | > +--+--+--+--+--+ > {code} > Cause: > In multi-serde parsing, the total length calculation here: > https://github.com/apache/hive/blob/rel/release-3.1.2/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java#L308 > does not take extra fields into account. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22360) MultiSerDe returns wrong results in last column when the loaded file has more columns than those in table schema
[ https://issues.apache.org/jira/browse/HIVE-22360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shubham Chaurasia updated HIVE-22360: - Description: Repro steps: Input file: {code} 1^,1^,^,0^,0^,0 2^,1^,^,0^,1^,0 3^,1^,^,0^,0^,0 4^,1^,^,0^,1^,0 {code} Queries: {code} CREATE TABLE n2(colA int, colB tinyint, colC timestamp, colD smallint, colE smallint) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.MultiDelimitSerDe' WITH SERDEPROPERTIES ("field.delim"="^,")STORED AS TEXTFILE; LOAD DATA LOCAL INPATH '/Users/schaurasia/Documents/input_6_cols.csv' OVERWRITE INTO TABLE n2; select * from n2; // wrong last column results here. +--+--+--+--+--+ | n2.cola | n2.colb | n2.colc | n2.cold | n2.cole | +--+--+--+--+--+ | 1| 1| NULL | 0| NULL | | 2| 1| NULL | 0| NULL | | 3| 1| NULL | 0| NULL | | 4| 1| NULL | 0| NULL | +--+--+--+--+--+ {code} Cause: In multi-serde parsing, the total length calculation here: https://github.com/apache/hive/blob/rel/release-3.1.2/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java#L308 does not take extra fields into account. was: Repro steps: Input file: {code} 1^,1^,^,0^,0^,0 2^,1^,^,0^,1^,0 3^,1^,^,0^,0^,0 4^,1^,^,0^,1^,0 {code} Queries: {code} CREATE TABLE n2(colA int, colB tinyint, colC timestamp, colD smallint, colE smallint) ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe' WITH SERDEPROPERTIES ("field.delim"="^,")STORED AS TEXTFILE; LOAD DATA LOCAL INPATH '/Users/schaurasia/Documents/input_6_cols.csv' OVERWRITE INTO TABLE n2; select * from n2; // wrong last column results here. +--+--+--+--+--+ | n2.cola | n2.colb | n2.colc | n2.cold | n2.cole | +--+--+--+--+--+ | 1| 1| NULL | 0| NULL | | 2| 1| NULL | 0| NULL | | 3| 1| NULL | 0| NULL | | 4| 1| NULL | 0| NULL | +--+--+--+--+--+ {code} Cause: In multi-serde parsing, the total length calculation here: https://github.com/apache/hive/blob/rel/release-3.1.2/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java#L308 does not take extra fields into account. > MultiSerDe returns wrong results in last column when the loaded file has more > columns than those in table schema > > > Key: HIVE-22360 > URL: https://issues.apache.org/jira/browse/HIVE-22360 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 4.0.0 >Reporter: Shubham Chaurasia >Assignee: Shubham Chaurasia >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22360.1.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Repro steps: > Input file: > {code} > 1^,1^,^,0^,0^,0 > 2^,1^,^,0^,1^,0 > 3^,1^,^,0^,0^,0 > 4^,1^,^,0^,1^,0 > {code} > Queries: > {code} > CREATE TABLE n2(colA int, colB tinyint, colC timestamp, colD smallint, colE > smallint) ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.MultiDelimitSerDe' > WITH SERDEPROPERTIES ("field.delim"="^,")STORED AS TEXTFILE; > LOAD DATA LOCAL INPATH '/Users/schaurasia/Documents/input_6_cols.csv' > OVERWRITE INTO TABLE n2; > select * from n2; > // wrong last column results here. > +--+--+--+--+--+ > | n2.cola | n2.colb | n2.colc | n2.cold | n2.cole | > +--+--+--+--+--+ > | 1| 1| NULL | 0| NULL | > | 2| 1| NULL | 0| NULL | > | 3| 1| NULL | 0| NULL | > | 4| 1| NULL | 0| NULL | > +--+--+--+--+--+ > {code} > Cause: > In multi-serde parsing, the total length calculation here: > https://github.com/apache/hive/blob/rel/release-3.1.2/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java#L308 > does not take extra fields into account. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22360) MultiSerDe returns wrong results in last column when the loaded file has more columns than those in table schema
[ https://issues.apache.org/jira/browse/HIVE-22360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shubham Chaurasia updated HIVE-22360: - Attachment: HIVE-22360.1.patch Status: Patch Available (was: Open) > MultiSerDe returns wrong results in last column when the loaded file has more > columns than those in table schema > > > Key: HIVE-22360 > URL: https://issues.apache.org/jira/browse/HIVE-22360 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 4.0.0 >Reporter: Shubham Chaurasia >Assignee: Shubham Chaurasia >Priority: Major > Labels: pull-request-available > Attachments: HIVE-22360.1.patch > > Time Spent: 10m > Remaining Estimate: 0h > > Repro steps: > Input file: > {code} > 1^,1^,^,0^,0^,0 > 2^,1^,^,0^,1^,0 > 3^,1^,^,0^,0^,0 > 4^,1^,^,0^,1^,0 > {code} > Queries: > {code} > CREATE TABLE n2(colA int, colB tinyint, colC timestamp, colD smallint, colE > smallint) ROW FORMAT SERDE > 'org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe' WITH > SERDEPROPERTIES ("field.delim"="^,")STORED AS TEXTFILE; > LOAD DATA LOCAL INPATH '/Users/schaurasia/Documents/input_6_cols.csv' > OVERWRITE INTO TABLE n2; > select * from n2; > // wrong last column results here. > +--+--+--+--+--+ > | n2.cola | n2.colb | n2.colc | n2.cold | n2.cole | > +--+--+--+--+--+ > | 1| 1| NULL | 0| NULL | > | 2| 1| NULL | 0| NULL | > | 3| 1| NULL | 0| NULL | > | 4| 1| NULL | 0| NULL | > +--+--+--+--+--+ > {code} > Cause: > In multi-serde parsing, the total length calculation here: > https://github.com/apache/hive/blob/rel/release-3.1.2/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java#L308 > does not take extra fields into account. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22360) MultiSerDe returns wrong results in last column when the loaded file has more columns than those in table schema
[ https://issues.apache.org/jira/browse/HIVE-22360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HIVE-22360: -- Labels: pull-request-available (was: ) > MultiSerDe returns wrong results in last column when the loaded file has more > columns than those in table schema > > > Key: HIVE-22360 > URL: https://issues.apache.org/jira/browse/HIVE-22360 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 4.0.0 >Reporter: Shubham Chaurasia >Assignee: Shubham Chaurasia >Priority: Major > Labels: pull-request-available > > Repro steps: > Input file: > {code} > 1^,1^,^,0^,0^,0 > 2^,1^,^,0^,1^,0 > 3^,1^,^,0^,0^,0 > 4^,1^,^,0^,1^,0 > {code} > Queries: > {code} > CREATE TABLE n2(colA int, colB tinyint, colC timestamp, colD smallint, colE > smallint) ROW FORMAT SERDE > 'org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe' WITH > SERDEPROPERTIES ("field.delim"="^,")STORED AS TEXTFILE; > LOAD DATA LOCAL INPATH '/Users/schaurasia/Documents/input_6_cols.csv' > OVERWRITE INTO TABLE n2; > select * from n2; > // wrong last column results here. > +--+--+--+--+--+ > | n2.cola | n2.colb | n2.colc | n2.cold | n2.cole | > +--+--+--+--+--+ > | 1| 1| NULL | 0| NULL | > | 2| 1| NULL | 0| NULL | > | 3| 1| NULL | 0| NULL | > | 4| 1| NULL | 0| NULL | > +--+--+--+--+--+ > {code} > Cause: > In multi-serde parsing, the total length calculation here: > https://github.com/apache/hive/blob/rel/release-3.1.2/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java#L308 > does not take extra fields into account. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22360) MultiSerDe returns wrong results in last column when the loaded file has more columns than those in table schema
[ https://issues.apache.org/jira/browse/HIVE-22360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shubham Chaurasia updated HIVE-22360: - Affects Version/s: (was: 3.1.2) 4.0.0 > MultiSerDe returns wrong results in last column when the loaded file has more > columns than those in table schema > > > Key: HIVE-22360 > URL: https://issues.apache.org/jira/browse/HIVE-22360 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 4.0.0 >Reporter: Shubham Chaurasia >Assignee: Shubham Chaurasia >Priority: Major > > Repro steps: > Input file: > {code} > 1^,1^,^,0^,0^,0 > 2^,1^,^,0^,1^,0 > 3^,1^,^,0^,0^,0 > 4^,1^,^,0^,1^,0 > {code} > Queries: > {code} > CREATE TABLE n2(colA int, colB tinyint, colC timestamp, colD smallint, colE > smallint) ROW FORMAT SERDE > 'org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe' WITH > SERDEPROPERTIES ("field.delim"="^,")STORED AS TEXTFILE; > LOAD DATA LOCAL INPATH '/Users/schaurasia/Documents/input_6_cols.csv' > OVERWRITE INTO TABLE n2; > select * from n2; > // wrong last column results here. > +--+--+--+--+--+ > | n2.cola | n2.colb | n2.colc | n2.cold | n2.cole | > +--+--+--+--+--+ > | 1| 1| NULL | 0| NULL | > | 2| 1| NULL | 0| NULL | > | 3| 1| NULL | 0| NULL | > | 4| 1| NULL | 0| NULL | > +--+--+--+--+--+ > {code} > Cause: > In multi-serde parsing, the total length calculation here: > https://github.com/apache/hive/blob/rel/release-3.1.2/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java#L308 > does not take extra fields into account. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (HIVE-22360) MultiSerDe returns wrong results in last column when the loaded file has more columns than those in table schema
[ https://issues.apache.org/jira/browse/HIVE-22360?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shubham Chaurasia updated HIVE-22360: - Description: Repro steps: Input file: {code} 1^,1^,^,0^,0^,0 2^,1^,^,0^,1^,0 3^,1^,^,0^,0^,0 4^,1^,^,0^,1^,0 {code} Queries: {code} CREATE TABLE n2(colA int, colB tinyint, colC timestamp, colD smallint, colE smallint) ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe' WITH SERDEPROPERTIES ("field.delim"="^,")STORED AS TEXTFILE; LOAD DATA LOCAL INPATH '/Users/schaurasia/Documents/input_6_cols.csv' OVERWRITE INTO TABLE n2; select * from n2; // wrong last column results here. +--+--+--+--+--+ | n2.cola | n2.colb | n2.colc | n2.cold | n2.cole | +--+--+--+--+--+ | 1| 1| NULL | 0| NULL | | 2| 1| NULL | 0| NULL | | 3| 1| NULL | 0| NULL | | 4| 1| NULL | 0| NULL | +--+--+--+--+--+ {code} Cause: In multi-serde parsing, the total length calculation here: https://github.com/apache/hive/blob/rel/release-3.1.2/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java#L308 does not take extra fields into account. was: Repro steps: Input file: {code} +--+ | 1^,1^,^,0^,0^,0 | | 2^,1^,^,0^,1^,0 | | 3^,1^,^,0^,0^,0 | | 4^,1^,^,0^,1^,0 | +--+ {code} Queries: {code} CREATE TABLE n2(colA int, colB tinyint, colC timestamp, colD smallint, colE smallint) ROW FORMAT SERDE 'org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe' WITH SERDEPROPERTIES ("field.delim"="^,")STORED AS TEXTFILE; LOAD DATA LOCAL INPATH '/Users/schaurasia/Documents/input_6_cols.csv' OVERWRITE INTO TABLE n2; select * from n2; // wrong last column results here. +--+--+--+--+--+ | n2.cola | n2.colb | n2.colc | n2.cold | n2.cole | +--+--+--+--+--+ | 1| 1| NULL | 0| NULL | | 2| 1| NULL | 0| NULL | | 3| 1| NULL | 0| NULL | | 4| 1| NULL | 0| NULL | +--+--+--+--+--+ {code} Cause: In multi-serde parsing, the total length calculation here: https://github.com/apache/hive/blob/rel/release-3.1.2/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java#L308 does not take extra fields into account. > MultiSerDe returns wrong results in last column when the loaded file has more > columns than those in table schema > > > Key: HIVE-22360 > URL: https://issues.apache.org/jira/browse/HIVE-22360 > Project: Hive > Issue Type: Bug > Components: Serializers/Deserializers >Affects Versions: 3.1.2 >Reporter: Shubham Chaurasia >Assignee: Shubham Chaurasia >Priority: Major > > Repro steps: > Input file: > {code} > 1^,1^,^,0^,0^,0 > 2^,1^,^,0^,1^,0 > 3^,1^,^,0^,0^,0 > 4^,1^,^,0^,1^,0 > {code} > Queries: > {code} > CREATE TABLE n2(colA int, colB tinyint, colC timestamp, colD smallint, colE > smallint) ROW FORMAT SERDE > 'org.apache.hadoop.hive.contrib.serde2.MultiDelimitSerDe' WITH > SERDEPROPERTIES ("field.delim"="^,")STORED AS TEXTFILE; > LOAD DATA LOCAL INPATH '/Users/schaurasia/Documents/input_6_cols.csv' > OVERWRITE INTO TABLE n2; > select * from n2; > // wrong last column results here. > +--+--+--+--+--+ > | n2.cola | n2.colb | n2.colc | n2.cold | n2.cole | > +--+--+--+--+--+ > | 1| 1| NULL | 0| NULL | > | 2| 1| NULL | 0| NULL | > | 3| 1| NULL | 0| NULL | > | 4| 1| NULL | 0| NULL | > +--+--+--+--+--+ > {code} > Cause: > In multi-serde parsing, the total length calculation here: > https://github.com/apache/hive/blob/rel/release-3.1.2/serde/src/java/org/apache/hadoop/hive/serde2/lazy/LazyStruct.java#L308 > does not take extra fields into account. -- This message was sent by Atlassian Jira (v8.3.4#803005)