[jira] [Commented] (SOLR-13622) Add FileStream Streaming Expression
[ https://issues.apache.org/jira/browse/SOLR-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902933#comment-16902933 ] Jason Gerlowski commented on SOLR-13622: The test issue should be resolved now; thanks for pointing it out Hoss. I'll close this in a few days if the test failures on Windows are truly resolved. > Add FileStream Streaming Expression > --- > > Key: SOLR-13622 > URL: https://issues.apache.org/jira/browse/SOLR-13622 > Project: Solr > Issue Type: New Feature > Components: streaming expressions >Reporter: Joel Bernstein >Assignee: Jason Gerlowski >Priority: Major > Fix For: 8.3 > > Attachments: SOLR-13622.patch, SOLR-13622.patch > > > The FileStream will read files from a local filesystem and Stream back each > line of the file as a tuple. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13622) Add FileStream Streaming Expression
[ https://issues.apache.org/jira/browse/SOLR-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902931#comment-16902931 ] ASF subversion and git services commented on SOLR-13622: Commit 299d92da5cc6315a98ef656a66ab7b285ecb4e3d in lucene-solr's branch refs/heads/branch_8x from Jason Gerlowski [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=299d92d ] SOLR-13622: Rename FilesStream -> CatStream Also fixes an 'cat' OS-dependent bug in StreamExpressionTest. > Add FileStream Streaming Expression > --- > > Key: SOLR-13622 > URL: https://issues.apache.org/jira/browse/SOLR-13622 > Project: Solr > Issue Type: New Feature > Components: streaming expressions >Reporter: Joel Bernstein >Assignee: Jason Gerlowski >Priority: Major > Fix For: 8.3 > > Attachments: SOLR-13622.patch, SOLR-13622.patch > > > The FileStream will read files from a local filesystem and Stream back each > line of the file as a tuple. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13622) Add FileStream Streaming Expression
[ https://issues.apache.org/jira/browse/SOLR-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902930#comment-16902930 ] ASF subversion and git services commented on SOLR-13622: Commit 2eb493d1700d59845ac120dcc485556b7e7fb422 in lucene-solr's branch refs/heads/master from Jason Gerlowski [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=2eb493d ] SOLR-13622: Rename FilesStream -> CatStream Also fixes an 'cat' OS-dependent bug in StreamExpressionTest. > Add FileStream Streaming Expression > --- > > Key: SOLR-13622 > URL: https://issues.apache.org/jira/browse/SOLR-13622 > Project: Solr > Issue Type: New Feature > Components: streaming expressions >Reporter: Joel Bernstein >Assignee: Jason Gerlowski >Priority: Major > Fix For: 8.3 > > Attachments: SOLR-13622.patch, SOLR-13622.patch > > > The FileStream will read files from a local filesystem and Stream back each > line of the file as a tuple. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13622) Add FileStream Streaming Expression
[ https://issues.apache.org/jira/browse/SOLR-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16902342#comment-16902342 ] Jason Gerlowski commented on SOLR-13622: Sorry, that's my mistake. Dumb mistake. I'll fix it right away. Joel, I'll do the rename while I'm at it. > Add FileStream Streaming Expression > --- > > Key: SOLR-13622 > URL: https://issues.apache.org/jira/browse/SOLR-13622 > Project: Solr > Issue Type: New Feature > Components: streaming expressions >Reporter: Joel Bernstein >Assignee: Jason Gerlowski >Priority: Major > Fix For: 8.3 > > Attachments: SOLR-13622.patch, SOLR-13622.patch > > > The FileStream will read files from a local filesystem and Stream back each > line of the file as a tuple. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13622) Add FileStream Streaming Expression
[ https://issues.apache.org/jira/browse/SOLR-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16900113#comment-16900113 ] Jason Gerlowski commented on SOLR-13622: That rename would be fine with me. I'm not strongly attached to the name. But if we change it we'll have to remember to update the ref-guide docs as well. Are you going to make this change or would you like me to? > Add FileStream Streaming Expression > --- > > Key: SOLR-13622 > URL: https://issues.apache.org/jira/browse/SOLR-13622 > Project: Solr > Issue Type: New Feature > Components: streaming expressions >Reporter: Joel Bernstein >Assignee: Jason Gerlowski >Priority: Major > Fix For: 8.3 > > Attachments: SOLR-13622.patch, SOLR-13622.patch > > > The FileStream will read files from a local filesystem and Stream back each > line of the file as a tuple. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13622) Add FileStream Streaming Expression
[
https://issues.apache.org/jira/browse/SOLR-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16899653#comment-16899653
]
Joel Bernstein commented on SOLR-13622:
---
The more that I test out this feature the less I like the function name
"files". I was thinking that a better name might be "cat".
The sample syntax would be:
{code:java}
cat("file.csv"){code}
> Add FileStream Streaming Expression
> ---
>
> Key: SOLR-13622
> URL: https://issues.apache.org/jira/browse/SOLR-13622
> Project: Solr
> Issue Type: New Feature
> Components: streaming expressions
>Reporter: Joel Bernstein
>Assignee: Jason Gerlowski
>Priority: Major
> Fix For: 8.3
>
> Attachments: SOLR-13622.patch, SOLR-13622.patch
>
>
> The FileStream will read files from a local filesystem and Stream back each
> line of the file as a tuple.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13622) Add FileStream Streaming Expression
[
https://issues.apache.org/jira/browse/SOLR-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16892736#comment-16892736
]
Jason Gerlowski commented on SOLR-13622:
I merged the initial version of this to {{master}} and {{branch_8x}}. I only
made two changes of note since my last post here:
* I changed the name of the expression from {{fileStream}} to {{files}}.
* I changed the chroot to be based out of a directory called
{{$SOLR_HOME/userfiles}}. The {{userfiles}} directory gets created if it
doesn't exist upon Solr startup and users can put files in after that point.
> Add FileStream Streaming Expression
> ---
>
> Key: SOLR-13622
> URL: https://issues.apache.org/jira/browse/SOLR-13622
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
> Components: streaming expressions
>Reporter: Joel Bernstein
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13622.patch, SOLR-13622.patch
>
>
> The FileStream will read files from a local filesystem and Stream back each
> line of the file as a tuple.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13622) Add FileStream Streaming Expression
[ https://issues.apache.org/jira/browse/SOLR-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16892727#comment-16892727 ] ASF subversion and git services commented on SOLR-13622: Commit fa9473df8feff74ddc2c00cf1d18b36e0899410b in lucene-solr's branch refs/heads/branch_8x from Jason Gerlowski [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=fa9473d ] SOLR-13622: Add fileStream stream-source > Add FileStream Streaming Expression > --- > > Key: SOLR-13622 > URL: https://issues.apache.org/jira/browse/SOLR-13622 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) > Components: streaming expressions >Reporter: Joel Bernstein >Assignee: Jason Gerlowski >Priority: Major > Attachments: SOLR-13622.patch, SOLR-13622.patch > > > The FileStream will read files from a local filesystem and Stream back each > line of the file as a tuple. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13622) Add FileStream Streaming Expression
[ https://issues.apache.org/jira/browse/SOLR-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16892723#comment-16892723 ] ASF subversion and git services commented on SOLR-13622: Commit dc8e9afff92f3ffc4081a2ecad5970eb09924a73 in lucene-solr's branch refs/heads/master from Jason Gerlowski [ https://gitbox.apache.org/repos/asf?p=lucene-solr.git;h=dc8e9af ] SOLR-13622: Add fileStream stream-source > Add FileStream Streaming Expression > --- > > Key: SOLR-13622 > URL: https://issues.apache.org/jira/browse/SOLR-13622 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) > Components: streaming expressions >Reporter: Joel Bernstein >Assignee: Jason Gerlowski >Priority: Major > Attachments: SOLR-13622.patch, SOLR-13622.patch > > > The FileStream will read files from a local filesystem and Stream back each > line of the file as a tuple. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13622) Add FileStream Streaming Expression
[
https://issues.apache.org/jira/browse/SOLR-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885162#comment-16885162
]
Joel Bernstein commented on SOLR-13622:
---
The StreamContext has a reference to the SolrCore where the request is being
executed.
{code:java}
SolrCore core = streamContext.get("solr-core"){code}
Once you have the core you can get the SolrResourceLoader and then figure out
how to fetch files.
The SolrCore is set for each request in StreamHandler line 174.
> Add FileStream Streaming Expression
> ---
>
> Key: SOLR-13622
> URL: https://issues.apache.org/jira/browse/SOLR-13622
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
> Components: streaming expressions
>Reporter: Joel Bernstein
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13622.patch
>
>
> The FileStream will read files from a local filesystem and Stream back each
> line of the file as a tuple.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13622) Add FileStream Streaming Expression
[
https://issues.apache.org/jira/browse/SOLR-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884805#comment-16884805
]
Jason Gerlowski commented on SOLR-13622:
I attached a quick-and-dirty POC for this. Right now you can invoke the
streaming expression as:
{{fileStream(/First/absolute/path|/Second/absolute/path}}
Some notes:
* Right now the filepath argument doesn't take quotes, and uses the pipe
character as a delimiter between args. Both of these are temporary things
we'll want to fix, they were just a bit easier to get working in the short term.
* Currently the path argument works with absolute paths. I did this because I
was blanking on how to figure out SOLR_HOME or SOLR_DATA_HOME from SolrJ code.
(The goto for this in solr-core is SolrResourceLoader, but that's not available
in SolrJ where all the streaming expressions are defined.) Maybe this isn't
possible from code that lives in SolrJ...going to have to think this through a
bit. If anyone knows a trick I'm missing, or a way around the problem, please
chime in.
* I haven't implemented the max-lines parameter yet, but that should be pretty
straightforward.
> Add FileStream Streaming Expression
> ---
>
> Key: SOLR-13622
> URL: https://issues.apache.org/jira/browse/SOLR-13622
> Project: Solr
> Issue Type: New Feature
> Security Level: Public(Default Security Level. Issues are Public)
> Components: streaming expressions
>Reporter: Joel Bernstein
>Assignee: Jason Gerlowski
>Priority: Major
> Attachments: SOLR-13622.patch
>
>
> The FileStream will read files from a local filesystem and Stream back each
> line of the file as a tuple.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org
For additional commands, e-mail: dev-h...@lucene.apache.org
[jira] [Commented] (SOLR-13622) Add FileStream Streaming Expression
[ https://issues.apache.org/jira/browse/SOLR-13622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884802#comment-16884802 ] Jason Gerlowski commented on SOLR-13622: Joel and I discussed this a bit offline and had some initial thoughts about what this should look like: * file specification could take either files or directories (which would then be processed recursively). Ideally the file parameter would allow a comma-delimited list of files/directories to process. * received filepaths would have to be evaluated relative to a specified particular data directory (to avoid the security issue of allowing reading arbitrary files on the Solr box). Also to this effect, we'd need to do some sanitizing of the file paths that users provide to ensure they're not escaping the sandbox we set up for them. * each emitted tuple could contain the filename/path of the file that the emitted tuple came from, to allow differentiation of lines from multiple files. * we could add a numeric parameter to cap the number of lines that get emitted if users just want to see the first N lines of a large file (or group of files) > Add FileStream Streaming Expression > --- > > Key: SOLR-13622 > URL: https://issues.apache.org/jira/browse/SOLR-13622 > Project: Solr > Issue Type: New Feature > Security Level: Public(Default Security Level. Issues are Public) > Components: streaming expressions >Reporter: Joel Bernstein >Assignee: Jason Gerlowski >Priority: Major > Attachments: SOLR-13622.patch > > > The FileStream will read files from a local filesystem and Stream back each > line of the file as a tuple. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: dev-unsubscr...@lucene.apache.org For additional commands, e-mail: dev-h...@lucene.apache.org
