Good to hear there will be partitioning support. I’ve had some success loading
partitioned data specified with Unix glowing format. i.e.:
sc.textFile("s3:/bucket/directory/dt=2014-11-{2[4-9],30}T00-00-00”)
would load dates 2014-11-24 through 2014-11-30. Not the most ideal solution,
but it seems to work for loading data from a range.
Best,
Chris
> On Jan 26, 2015, at 10:55 AM, Cheng Lian <[email protected]> wrote:
>
> Currently no if you don't want to use Spark SQL's HiveContext. But we're
> working on adding partitioning support to the external data sources API, with
> which you can create, for example, partitioned Parquet tables without using
> Hive.
>
> Cheng
>
> On 1/26/15 8:47 AM, Danny Yates wrote:
>> Thanks Michael.
>>
>> I'm not actually using Hive at the moment - in fact, I'm trying to avoid it
>> if I can. I'm just wondering whether Spark has anything similar I can
>> leverage?
>>
>> Thanks
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: [email protected]
> For additional commands, e-mail: [email protected]
>
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
