AngersZhuuuu edited a comment on pull request #29421:
URL: https://github.com/apache/spark/pull/29421#issuecomment-677495619
Hive serde mode schema less behavior
Input column size >2
```
> select transform(*)
> using 'cat'
> from (
> select 1 as a, 2 as b, 3 as c
> ) tmp ;
OK
1 2 3
Time taken: 25.465 seconds, Fetched: 1 row(s)
```
Input column size >2 and specify serde
```
> select transform(*)
>. ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
> using 'cat'
> ROW FORMAT SERDE 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
> from (
> select 1 as a, 2 as b, 3 as c
> ) tmp ;
OK
1 2
Time taken: 25.465 seconds, Fetched: 1 row(s)
```
Input column size > 2 specify serde with
`'serialization.last.column.takes.rest' = 'true'`
```
> select transform(*)
> ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
> WITH SERDEPROPERTIES (
> 'serialization.last.column.takes.rest' = 'true'
> )
> using 'cat'
> ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
> WITH SERDEPROPERTIES (
> 'serialization.last.column.takes.rest' = 'true'
> )
> from (
> select 1 as a, 2 as b, 3 as c
> ) tmp ;
OK
1 2 3
Time taken: 24.832 seconds, Fetched: 1 row(s)
```
Input column size > 2 specify serde with
`'serialization.last.column.takes.rest' = 'false'`
```
hive>
>
> select transform(*)
> ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
> WITH SERDEPROPERTIES (
> 'serialization.last.column.takes.rest' = 'false'
> )
> using 'cat'
> ROW FORMAT SERDE
'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
> WITH SERDEPROPERTIES (
> 'serialization.last.column.takes.rest' = 'false'
> )
> from (
> select 1 as a, 2 as b, 3 as c
> ) tmp ;
OK
1 2
Time taken: 25.921 seconds, Fetched: 1 row(s)
hive>
```
Input column size = 2
```
hive> select transform(*)
> using 'cat'
> from (
> select 1 as a, 2 as b
> ) tmp ;
OK
1 2
Time taken: 27.217 seconds, Fetched: 1 row(s)
```
input column < 2
```
hive> select transform(*)
> using 'cat'
> from (
> select 1 as a
> ) tmp ;
OK
1 NULL
Time taken: 23.717 seconds, Fetched: 1 row(s)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
