Hi John,I've been following your trail of emails :) Thanks for sharing all that info, it's very useful.
I think I'm trying to do something very similar to what you did. I have data flowing from Storm into S3 and I wanted to be able to periodically preprocess/repartition into new folder and then have views to merge recent data from the raw Storm files and old data from the preprocessed/repartitioned folders. These views are intended to be used from Tableau.
I guess I can create a small process that checks when a new folder with preprocessed data is available and replaces the appropriate view files with new versions that have the proper date string. But it would be a lot nicer to just do it in the view and have a dumb process executing the periodic queries.
How did you "solve" it in the end? If I can ask. Thanks, Oscar On Wed, Jul 20, 2016 at 07:26:20AM -0500, John Omernik wrote:
I think I ran into that issue before and (someone will correct me if I am wrong) the issue is that current_date is only materialized AFTER planning. Thus the pruning, which occurs during planning doesn't happen. Is this a programatic query or just something that is being done for users? I know my issue was I wanted a view that showed only the current date, and I struggled to come up with a good solution to that. John On Wed, Jul 20, 2016 at 6:06 AM, Oscar Morante <[email protected]> wrote:I'm trying to trigger partition prunning like this: select * from dfs.`json/by-date` where dir0 = cast(current_date as varchar); But apparently, it only works when passing a literal. Am I missing something? Thanks,
signature.asc
Description: Digital signature
