I was unable to get this feature to work in 1.3.0. I tried building off master
and it still wasn't working for me. So I dug into the code, and I'm not sure
how the parsePartition() was ever working. The while loop which walks up the
parent directories in the path always terminates after a single iteration. I
made a minor change and the partition discovery appears to work now.
Specifically, I changed
var chopped = path
while (!finished) {
val maybeColumn = parsePartitionColumn(chopped.getName,
defaultPartitionName)
maybeColumn.foreach(columns += _)
chopped = chopped.getParent
finished = maybeColumn.isEmpty || chopped.getParent == null
}
To
var chopped = path
while (chopped != null) {
val maybeColumn = parsePartitionColumn(chopped.getName,
defaultPartitionName)
maybeColumn.foreach(columns += _)
chopped = chopped.getParent
}
Because the leaf nodes are always named data.parquet, this loop was terminating
immediately after the first iteration. The only other thought I had is that the
loop may have been intended to walk up the path until it stopped finding
partition directories. In this case, the loop would work fine as is, but
chopped should be initialized to path.getParent rather than path.
I'm completely new to spark so it's possible that I misunderstood the intent
here completely, but if not then I'm happy to open an issue and submit a pull
request for whichever approach is the correct one.
This e-mail and its attachments are intended only for the individual or entity
to whom it is addressed and may contain information that is confidential,
privileged, inside information, or subject to other restrictions on use or
disclosure. Any unauthorized use, dissemination or copying of this transmission
or the information in it is prohibited and may be unlawful. If you have
received this transmission in error, please notify the sender immediately by
return e-mail, and permanently delete or destroy this e-mail, any attachments,
and all copies (digital or paper). Unless expressly stated in this e-mail,
nothing in this message should be construed as a digital or electronic
signature. For additional important disclaimers and disclosures regarding KCG's
products and services, please click on the following link:
http://www.kcg.com/legal/global-disclosures
