Hi Ryan,

The attachment is the event timeline on executors. They are always busy 
computing.

More executors are helpful but that's not my job as a developer.


1. The bad performance could be caused by my poor implementation, as "checkID" 
would not pushdown as a user defined function.

2. To make the group index works, I need to sort the data by id, which leads to 
shuffle of 50T data. That's somehow crazy.


I'm on the way testing HAR, but the discussion brings me lots of insight about 
ORC.

Thanks for your help!


________________________________
发件人: Ryan <ryan.hd....@gmail.com>
发送时间: 2017年4月17日 16:48:47
收件人: 莫涛
抄送: user
主题: Re: 答复: 答复: How to store 10M records in HDFS to speed up further filtering?

how about the event timeline on executors? It seems add more executor could 
help.

1. I found a jira(https://issues.apache.org/jira/browse/SPARK-11621) that 
states the ppd should work. And I think "only for matched ones the binary data 
is read" is true if proper index is configured. The row group wouldn't be read 
if the predicate isn't satisfied due to index.

2. It is absolutely true the performance gain depends on the id distribution...

On Mon, Apr 17, 2017 at 4:23 PM, 莫涛 
<mo...@sensetime.com<mailto:mo...@sensetime.com>> wrote:

Hi Ryan,


The attachment is a screen shot for the spark job and this is the only stage 
for this job.

I've changed the partition size to 1GB by "--conf 
spark.sql.files.maxPartitionBytes=1073741824<tel:010%207374%201824>".


1. spark-orc seems not that smart. The input size is almost the whole data. I 
guess "only for matched ones the binary data is read" is not true as orc does 
not know the offset of each BINARY so things like seek could not happen

2. I've tried orc and it does skip the partition that has no hit. This could be 
a solution but the performance depends on the distribution of the given ID 
list. No partition could be skipped in the worst case.


Mo Tao



________________________________
发件人: Ryan <ryan.hd....@gmail.com<mailto:ryan.hd....@gmail.com>>
发送时间: 2017年4月17日 15:42:46
收件人: 莫涛
抄送: user
主题: Re: 答复: How to store 10M records in HDFS to speed up further filtering?

1. Per my understanding, for orc files, it should push down the filters, which 
means all id columns will be scanned but only for matched ones the binary data 
is read. I haven't dig into spark-orc reader though..

2. orc itself have row group index and bloom filter index. you may try 
configurations like 'orc.bloom.filter.columns' on the source table first. From 
the spark side, with mapPartitions, it's possible to build sort of index for 
each partition.

And could you check how many tasks does the filter stage have? maybe there's 
too few partitions..

On Mon, Apr 17, 2017 at 3:01 PM, 莫涛 
<mo...@sensetime.com<mailto:mo...@sensetime.com>> wrote:

Hi Ryan,


1. "expected qps and response time for the filter request"

I expect that only the requested BINARY are scanned instead of all records, so 
the response time would be "10K * 5MB / disk read speed", or several times of 
this.

In practice, our cluster has 30 SAS disks and scanning all the 10M * 5MB data 
takes about 6 hours now. It should becomes several minutes as expected.


2. "build a search tree using ids within each partition to act like an index, 
or create a bloom filter to see if current partition would have any hit"

Sounds like the thing I'm looking for!

Could you kindly provide some links for reference? I found nothing in spark 
document about index or bloom filter working inside partition.


Thanks very much!


Mo Tao

________________________________
发件人: Ryan <ryan.hd....@gmail.com<mailto:ryan.hd....@gmail.com>>
发送时间: 2017年4月17日 14:32:00
收件人: 莫涛
抄送: user
主题: Re: How to store 10M records in HDFS to speed up further filtering?

you can build a search tree using ids within each partition to act like an 
index, or create a bloom filter to see if current partition would have any hit.

What's your expected qps and response time for the filter request?


On Mon, Apr 17, 2017 at 2:23 PM, MoTao 
<mo...@sensetime.com<mailto:mo...@sensetime.com>> wrote:
Hi all,

I have 10M (ID, BINARY) record, and the size of each BINARY is 5MB on
average.
In my daily application, I need to filter out 10K BINARY according to an ID
list.
How should I store the whole data to make the filtering faster?

I'm using DataFrame in Spark 2.0.0 and I've tried row-based format (avro)
and column-based format (orc).
However, both of them require to scan almost ALL records, making the
filtering stage very very slow.
The code block for filtering looks like:

val IDSet: Set[String] = ...
val checkID = udf { ID: String => IDSet(ID) }
spark.read.orc("/path/to/whole/data")
  .filter(checkID($"ID"))
  .select($"ID", $"BINARY")
  .write...

Thanks for any advice!




--
View this message in context: 
http://apache-spark-user-list.1001560.n3.nabble.com/How-to-store-10M-records-in-HDFS-to-speed-up-further-filtering-tp28605.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.

---------------------------------------------------------------------
To unsubscribe e-mail: 
user-unsubscr...@spark.apache.org<mailto:user-unsubscr...@spark.apache.org>




---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscr...@spark.apache.org

Reply via email to