betodealmeida commented on a change in pull request #4724: Improve database
type inference
URL:
https://github.com/apache/incubator-superset/pull/4724#discussion_r178584591
##########
File path: superset/dataframe.py
##########
@@ -41,26 +63,50 @@ class SupersetDataFrame(object):
'V': None, # raw data (void)
}
- def __init__(self, df):
- self.__df = df.where((pd.notnull(df)), None)
+ def __init__(self, data, cursor_description, db_engine_spec):
+ column_names = []
+ if cursor_description:
+ column_names = [col[0] for col in cursor_description]
+ self.column_names = dedup(column_names)
+
+ # check whether the result set has any nested dict columns
+ if data:
+ has_dict_col = any([isinstance(c, dict) for c in data[0]])
+ df_data = list(data) if has_dict_col else np.array(data,
dtype=object)
Review comment:
I tested the behavior with `[(1, 2, {'a': 'b'})]` as data, trying to
understand the code. If we build the dataframe using `np.array` on data
**with** a dict we get the following types for the columns:
```python
>>> data = [(1, 2, {'a': 'b'})]
>>> df = pd.DataFrame(np.array(data, dtype=object))
>>> print(df.dtypes)
0 object
1 object
2 object
dtype: object
```
Here Pandas doesn't know that the first two columns are integers. OTOH,
using `list(data)`:
```python
>>> data = [(1, 2, {'a': 'b'})]
>>> df = pd.DataFrame(list(data))
>>> print(df.dtypes)
0 int64
1 int64
2 object
dtype: object
```
Now we have correct type information for all columns. So it seems to me that
the `list(data) if has_dict_col else np.array(data, dtype=object)` is trying to
preserve the type information.
We can get the same result using `infer_objects()`:
```python
>>> data = [(1, 2, {'a': 'b'})]
>>> df = pd.DataFrame(np.array(data, dtype=object)).infer_objects()
>>> print(df.dtypes)
0 int64
1 int64
2 object
dtype: object
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
With regards,
Apache Git Services
