[
https://issues.apache.org/jira/browse/ARROW-2079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17125900#comment-17125900
]
Martin Durant commented on ARROW-2079:
--
Indeed, I believe spark always writes _commo
[
https://issues.apache.org/jira/browse/ARROW-2079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17116728#comment-17116728
]
Martin Durant commented on ARROW-2079:
--
`to_parquet` currently has `write_metadata_f
[
https://issues.apache.org/jira/browse/ARROW-2079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17116721#comment-17116721
]
Martin Durant commented on ARROW-2079:
--
> why the need of 2 files
Exactly - the
[
https://issues.apache.org/jira/browse/ARROW-5156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17105408#comment-17105408
]
Martin Durant commented on ARROW-5156:
--
Note that with something like [https://githu
[
https://issues.apache.org/jira/browse/ARROW-3247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17047692#comment-17047692
]
Martin Durant commented on ARROW-3247:
--
The spark-type maps:
{code:java}
- spark_
[
https://issues.apache.org/jira/browse/ARROW-3247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17045910#comment-17045910
]
Martin Durant commented on ARROW-3247:
--
They are allowed by the parquet spec, but no
[
https://issues.apache.org/jira/browse/ARROW-5072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16931552#comment-16931552
]
Martin Durant commented on ARROW-5072:
--
Ideally, you should write within a context l
[
https://issues.apache.org/jira/browse/ARROW-5156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16927617#comment-16927617
]
Martin Durant commented on ARROW-5156:
--
Note that it is the intent of the filesystem
[
https://issues.apache.org/jira/browse/ARROW-453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907592#comment-16907592
]
Martin Durant commented on ARROW-453:
-
> Turi Create (formerly Dato Create) accepts fu
[
https://issues.apache.org/jira/browse/ARROW-453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907592#comment-16907592
]
Martin Durant edited comment on ARROW-453 at 8/14/19 8:05 PM:
--
[
https://issues.apache.org/jira/browse/ARROW-453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907589#comment-16907589
]
Martin Durant commented on ARROW-453:
-
Yes, I mean the whole spec, so that you make ge
[
https://issues.apache.org/jira/browse/ARROW-453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907582#comment-16907582
]
Martin Durant commented on ARROW-453:
-
Can you *please* follow the fsspec model? I thi
[
https://issues.apache.org/jira/browse/ARROW-453?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16907571#comment-16907571
]
Martin Durant commented on ARROW-453:
-
Obviously, I choose to do this one particular w
[
https://issues.apache.org/jira/browse/ARROW-5156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845979#comment-16845979
]
Martin Durant commented on ARROW-5156:
--
Happy to add `_isfilestore` to s3fs/fsspec -
[
https://issues.apache.org/jira/browse/ARROW-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845947#comment-16845947
]
Martin Durant commented on ARROW-5349:
--
> in which this would be wrong if it is ins
[
https://issues.apache.org/jira/browse/ARROW-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845920#comment-16845920
]
Martin Durant commented on ARROW-5349:
--
It depends on what is passed back to the cal
[
https://issues.apache.org/jira/browse/ARROW-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845886#comment-16845886
]
Martin Durant commented on ARROW-5349:
--
> I think it's acceptable to set the path in
[
https://issues.apache.org/jira/browse/ARROW-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845885#comment-16845885
]
Martin Durant commented on ARROW-5349:
--
Agreed on that last point, to let the caller
[
https://issues.apache.org/jira/browse/ARROW-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16845871#comment-16845871
]
Martin Durant commented on ARROW-5349:
--
No, I don't have an explicit reference for t
[
https://issues.apache.org/jira/browse/ARROW-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16826179#comment-16826179
]
Martin Durant commented on ARROW-1983:
--
I don't know about deprecated, and I wouldn'
[
https://issues.apache.org/jira/browse/ARROW-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16826178#comment-16826178
]
Martin Durant commented on ARROW-1983:
--
I don't know about deprecated, and I wouldn'
[
https://issues.apache.org/jira/browse/ARROW-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16826171#comment-16826171
]
Martin Durant commented on ARROW-1983:
--
I don't know about "deprecated" (or whether
[
https://issues.apache.org/jira/browse/ARROW-5156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16823092#comment-16823092
]
Martin Durant commented on ARROW-5156:
--
I wasn't involved in the pandas code here. T
[
https://issues.apache.org/jira/browse/ARROW-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16820090#comment-16820090
]
Martin Durant commented on ARROW-1983:
--
> If readers would be able to use metadata f
[
https://issues.apache.org/jira/browse/ARROW-5144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16819005#comment-16819005
]
Martin Durant commented on ARROW-5144:
--
Well this is confusing! We don't explicitly
[
https://issues.apache.org/jira/browse/ARROW-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818370#comment-16818370
]
Martin Durant commented on ARROW-1983:
--
> Note that the Parquet format has three dif
[
https://issues.apache.org/jira/browse/ARROW-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818370#comment-16818370
]
Martin Durant edited comment on ARROW-1983 at 4/15/19 8:40 PM:
[
https://issues.apache.org/jira/browse/ARROW-1983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818334#comment-16818334
]
Martin Durant commented on ARROW-1983:
--
A convention, yes, but not in the parquet st
[
https://issues.apache.org/jira/browse/ARROW-5144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16818331#comment-16818331
]
Martin Durant commented on ARROW-5144:
--
Cloudpickle actually does a better job than
Martin Durant created ARROW-5144:
Summary: ParquetDataset and CloudParuqtePiece not serializable
Key: ARROW-5144
URL: https://issues.apache.org/jira/browse/ARROW-5144
Project: Apache Arrow
Is
[
https://issues.apache.org/jira/browse/ARROW-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16631203#comment-16631203
]
Martin Durant commented on ARROW-3238:
--
Thanks for that detail, Wes. Not sure when I
[
https://issues.apache.org/jira/browse/ARROW-3245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16621990#comment-16621990
]
Martin Durant commented on ARROW-3245:
--
(pyarrow 0.10.0)
```
In [7]: df = pd.DataF
[
https://issues.apache.org/jira/browse/ARROW-3245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16621400#comment-16621400
]
Martin Durant commented on ARROW-3245:
--
Sorry to be thick, but how do I actually get
[
https://issues.apache.org/jira/browse/ARROW-3245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16621009#comment-16621009
]
Martin Durant commented on ARROW-3245:
--
Could you please point me to documentation o
[
https://issues.apache.org/jira/browse/ARROW-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16617003#comment-16617003
]
Martin Durant commented on ARROW-3246:
--
> can only be implemented in the narrow case
[
https://issues.apache.org/jira/browse/ARROW-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Durant updated ARROW-3246:
-
Priority: Minor (was: Major)
> direct reading/writing of pandas categoricals in parquet
> --
[
https://issues.apache.org/jira/browse/ARROW-3245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Durant updated ARROW-3245:
-
Description:
The metadata included in parquet generally gives the min/max of data for each
chunk
[
https://issues.apache.org/jira/browse/ARROW-3247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Durant updated ARROW-3247:
-
Summary: Support spark parquet array and map types (was: Support spark
array and map types)
> S
[
https://issues.apache.org/jira/browse/ARROW-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Martin Durant updated ARROW-3246:
-
Summary: direct reading/writing of pandas categoricals in parquet (was:
direct reading/writing o
Martin Durant created ARROW-3247:
Summary: Support spark array and map types
Key: ARROW-3247
URL: https://issues.apache.org/jira/browse/ARROW-3247
Project: Apache Arrow
Issue Type: Improvemen
Martin Durant created ARROW-3246:
Summary: direct reading/writing of pandas categoricals
Key: ARROW-3246
URL: https://issues.apache.org/jira/browse/ARROW-3246
Project: Apache Arrow
Issue Type
Martin Durant created ARROW-3245:
Summary: Infer index and/or filtering from parquet column
statistics
Key: ARROW-3245
URL: https://issues.apache.org/jira/browse/ARROW-3245
Project: Apache Arrow
Martin Durant created ARROW-3244:
Summary: Multi-file parquet loading without scan
Key: ARROW-3244
URL: https://issues.apache.org/jira/browse/ARROW-3244
Project: Apache Arrow
Issue Type: Impr
[
https://issues.apache.org/jira/browse/ARROW-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16616303#comment-16616303
]
Martin Durant commented on ARROW-3238:
--
Doesn't
`use_dictionary=False`
preclude th
[
https://issues.apache.org/jira/browse/ARROW-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16150824#comment-16150824
]
Martin Durant commented on ARROW-1445:
--
Certainly not while at the same time remainin
[
https://issues.apache.org/jira/browse/ARROW-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16150702#comment-16150702
]
Martin Durant commented on ARROW-1445:
--
Not any easy solution: the data passed back i
[
https://issues.apache.org/jira/browse/ARROW-1445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16150523#comment-16150523
]
Martin Durant commented on ARROW-1445:
--
So sorry, I should have let you know I was wo
[
https://issues.apache.org/jira/browse/ARROW-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16111211#comment-16111211
]
Martin Durant commented on ARROW-1319:
--
Methods that I don't think exist, and some th
Martin Durant created ARROW-1322:
Summary: hdfs: encryption-at-rest and secure transport
Key: ARROW-1322
URL: https://issues.apache.org/jira/browse/ARROW-1322
Project: Apache Arrow
Issue Type
Martin Durant created ARROW-1321:
Summary: hdfs delegation token functions
Key: ARROW-1321
URL: https://issues.apache.org/jira/browse/ARROW-1321
Project: Apache Arrow
Issue Type: Improvement
Martin Durant created ARROW-1320:
Summary: hdfs block locations
Key: ARROW-1320
URL: https://issues.apache.org/jira/browse/ARROW-1320
Project: Apache Arrow
Issue Type: Improvement
Martin Durant created ARROW-1319:
Summary: hdfs methods
Key: ARROW-1319
URL: https://issues.apache.org/jira/browse/ARROW-1319
Project: Apache Arrow
Issue Type: Improvement
Reporte
Martin Durant created ARROW-1318:
Summary: hdfs access with auth
Key: ARROW-1318
URL: https://issues.apache.org/jira/browse/ARROW-1318
Project: Apache Arrow
Issue Type: Test
Repor
Martin Durant created ARROW-1317:
Summary: hdfs environment variables
Key: ARROW-1317
URL: https://issues.apache.org/jira/browse/ARROW-1317
Project: Apache Arrow
Issue Type: Improvement
Martin Durant created ARROW-1316:
Summary: hdfs connector stand-alone
Key: ARROW-1316
URL: https://issues.apache.org/jira/browse/ARROW-1316
Project: Apache Arrow
Issue Type: Wish
[
https://issues.apache.org/jira/browse/ARROW-1314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16111041#comment-16111041
]
Martin Durant commented on ARROW-1314:
--
It is the general distribution, e.g.,
http:/
[
https://issues.apache.org/jira/browse/ARROW-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16111018#comment-16111018
]
Martin Durant commented on ARROW-1313:
--
That would install the whole of hadoop as sys
Martin Durant created ARROW-1314:
Summary: libhdfs installation didn't work - mac
Key: ARROW-1314
URL: https://issues.apache.org/jira/browse/ARROW-1314
Project: Apache Arrow
Issue Type: Impro
[
https://issues.apache.org/jira/browse/ARROW-1313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16110960#comment-16110960
]
Martin Durant commented on ARROW-1313:
--
Docker file:
https://github.com/dask/hdfs3/b
Martin Durant created ARROW-1313:
Summary: libhdfs installation didn't work
Key: ARROW-1313
URL: https://issues.apache.org/jira/browse/ARROW-1313
Project: Apache Arrow
Issue Type: Improvement
[
https://issues.apache.org/jira/browse/ARROW-1293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16109813#comment-16109813
]
Martin Durant commented on ARROW-1293:
--
This very likely comes from snappy for the sp
61 matches
Mail list logo
