Github user HyukjinKwon commented on the pull request:
https://github.com/apache/spark/pull/11724#issuecomment-202220872
The commits I just added include the behaviour below:
#### Infering Types
- `DecimalType` is tried first. So, `10.1` (scale < precision) and numbers
with a precision less than 38 will be inferred as `DecimalType`.
- If it fails, then it uses `DoubleType`. For example, `0.01` (scale >
precision) or numbers with a precision bigger than 38 will be inferred
`DoubleType`.
#### Compatible Types (Merging Types)
- If inferred types in the same field during merging `DataType`s are
`DoubleType` and `DecimalType`,
`DoubleType` has higher priority.
- If inferred types in the same field during merging `DataType`s are
`DecimalType` and `DecimalType`, `DecimalType` will be created if it is
capable. If not, it falls back to `DoubleType`.
- If inferred types in the same field during merging `DataType`s are
`IntegralType` and `DecimalType`, `DecimalType` will be created if it is
capable. If not, it falls back to `DoubleType`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at [email protected] or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
