[jira] [Commented] (BEAM-2460) Make python maven variables available elsewhere
[ https://issues.apache.org/jira/browse/BEAM-2460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056727#comment-16056727 ] Ahmet Altay commented on BEAM-2460: --- Sounds good to me, thank you for the detailed explanation. > Make python maven variables available elsewhere > --- > > Key: BEAM-2460 > URL: https://issues.apache.org/jira/browse/BEAM-2460 > Project: Beam > Issue Type: Task > Components: sdk-java-core, sdk-py >Reporter: Stephen Sisk >Assignee: Stephen Sisk >Priority: Minor > > We're planning to start using perfkit to make it easy for users to run the IO > ITs. Perfkit is a python app, so we'll need to invoke the python interpreter. > findSupportedPython calculates this and sets the python.interpreter.bin > variable. It currently does so in the python project. > I believe the correct answer here is to move findSupportedPython so that's > it's invoked in the root pom. (but perhaps there's a more maven-y approach to > this problem?) > cc [~altay] [~markflyhigh] [~jasonkuster] [~davor] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BEAM-2460) Make python maven variables available elsewhere
[ https://issues.apache.org/jira/browse/BEAM-2460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056682#comment-16056682 ] Stephen Sisk commented on BEAM-2460: You are correct, that does mean that a project that wants to use python has have a hardcoded string (project.parent.parent...basedir) to know how to find the root pom. I think the solution I propose here is the best solution I know of because: 1. It doesn't require modifying every pom file 2. It doesn't require adding 2 copies of the same groovy code to the root pom file 3. It should allow for easy copy pasting between io projects that "just works" 4. If a project is going to break because of this change, it'll be a project where the code snippet has been added, giving people a fighting chance at debugging the error. If someone from the community has a suggestion on a better solution, I'm all ears, but based on the above stackoverflow discussions, I believe this is the best one for the particular problem that we have. > Make python maven variables available elsewhere > --- > > Key: BEAM-2460 > URL: https://issues.apache.org/jira/browse/BEAM-2460 > Project: Beam > Issue Type: Task > Components: sdk-java-core, sdk-py >Reporter: Stephen Sisk >Assignee: Stephen Sisk >Priority: Minor > > We're planning to start using perfkit to make it easy for users to run the IO > ITs. Perfkit is a python app, so we'll need to invoke the python interpreter. > findSupportedPython calculates this and sets the python.interpreter.bin > variable. It currently does so in the python project. > I believe the correct answer here is to move findSupportedPython so that's > it's invoked in the root pom. (but perhaps there's a more maven-y approach to > this problem?) > cc [~altay] [~markflyhigh] [~jasonkuster] [~davor] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BEAM-2460) Make python maven variables available elsewhere
[
https://issues.apache.org/jira/browse/BEAM-2460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056615#comment-16056615
]
Ahmet Altay commented on BEAM-2460:
---
I am not a Maven expert and feel free to ignore this. How does
{{{project.parent.parent.parent.parent.basedir}/sdks/python/findSupportedPython.groovy}}
work? Does not it require an arbitrary project to reference some source file
relative to root?
> Make python maven variables available elsewhere
> ---
>
> Key: BEAM-2460
> URL: https://issues.apache.org/jira/browse/BEAM-2460
> Project: Beam
> Issue Type: Task
> Components: sdk-java-core, sdk-py
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
>Priority: Minor
>
> We're planning to start using perfkit to make it easy for users to run the IO
> ITs. Perfkit is a python app, so we'll need to invoke the python interpreter.
> findSupportedPython calculates this and sets the python.interpreter.bin
> variable. It currently does so in the python project.
> I believe the correct answer here is to move findSupportedPython so that's
> it's invoked in the root pom. (but perhaps there's a more maven-y approach to
> this problem?)
> cc [~altay] [~markflyhigh] [~jasonkuster] [~davor]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BEAM-2460) Make python maven variables available elsewhere
[ https://issues.apache.org/jira/browse/BEAM-2460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056609#comment-16056609 ] Stephen Sisk commented on BEAM-2460: I wish that would help but it doesn't. The fundamental problem is that arbitrary projects can't easily reference the root project's directory. > Make python maven variables available elsewhere > --- > > Key: BEAM-2460 > URL: https://issues.apache.org/jira/browse/BEAM-2460 > Project: Beam > Issue Type: Task > Components: sdk-java-core, sdk-py >Reporter: Stephen Sisk >Assignee: Stephen Sisk >Priority: Minor > > We're planning to start using perfkit to make it easy for users to run the IO > ITs. Perfkit is a python app, so we'll need to invoke the python interpreter. > findSupportedPython calculates this and sets the python.interpreter.bin > variable. It currently does so in the python project. > I believe the correct answer here is to move findSupportedPython so that's > it's invoked in the root pom. (but perhaps there's a more maven-y approach to > this problem?) > cc [~altay] [~markflyhigh] [~jasonkuster] [~davor] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BEAM-2460) Make python maven variables available elsewhere
[
https://issues.apache.org/jira/browse/BEAM-2460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056558#comment-16056558
]
Stephen Sisk commented on BEAM-2460:
With a bit more investigation, I've landed on the following solution:
We will copy the groovy-maven-plugin invocation into the poms that need it, and
use project.parent.parent.[...].basedir to create a path that points at
findSupportedPython.groovy
So for the jdbc pom, this looks like:
org.codehaus.gmaven
groovy-maven-plugin
${groovy-maven-plugin.version}
find-supported-python-for-compile
initialize
execute
${project.parent.parent.parent.parent.basedir}/sdks/python/findSupportedPython.groovy
Where the project.parent.parent.parent... is specific to the directory that
it's in.
Conveniently, this will be the same for all the poms, so copy-pasting it will
work seamlessly with the io poms. I won't say I'm a big fan of this solution,
but given the alternatives, this seems like the least horrible one.
> Make python maven variables available elsewhere
> ---
>
> Key: BEAM-2460
> URL: https://issues.apache.org/jira/browse/BEAM-2460
> Project: Beam
> Issue Type: Task
> Components: sdk-java-core, sdk-py
>Reporter: Stephen Sisk
>Assignee: Stephen Sisk
>Priority: Minor
>
> We're planning to start using perfkit to make it easy for users to run the IO
> ITs. Perfkit is a python app, so we'll need to invoke the python interpreter.
> findSupportedPython calculates this and sets the python.interpreter.bin
> variable. It currently does so in the python project.
> I believe the correct answer here is to move findSupportedPython so that's
> it's invoked in the root pom. (but perhaps there's a more maven-y approach to
> this problem?)
> cc [~altay] [~markflyhigh] [~jasonkuster] [~davor]
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (BEAM-2460) Make python maven variables available elsewhere
[ https://issues.apache.org/jira/browse/BEAM-2460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056401#comment-16056401 ] Stephen Sisk commented on BEAM-2460: I realized that #2 above implies that I'd have to add a variable to every single project in the repo, that doesn't seem acceptable. > Make python maven variables available elsewhere > --- > > Key: BEAM-2460 > URL: https://issues.apache.org/jira/browse/BEAM-2460 > Project: Beam > Issue Type: Task > Components: sdk-java-core, sdk-py >Reporter: Stephen Sisk >Assignee: Stephen Sisk >Priority: Minor > > We're planning to start using perfkit to make it easy for users to run the IO > ITs. Perfkit is a python app, so we'll need to invoke the python interpreter. > findSupportedPython calculates this and sets the python.interpreter.bin > variable. It currently does so in the python project. > I believe the correct answer here is to move findSupportedPython so that's > it's invoked in the root pom. (but perhaps there's a more maven-y approach to > this problem?) > cc [~altay] [~markflyhigh] [~jasonkuster] [~davor] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BEAM-2460) Make python maven variables available elsewhere
[ https://issues.apache.org/jira/browse/BEAM-2460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056396#comment-16056396 ] Stephen Sisk commented on BEAM-2460: okay, this is more complex than I had hoped.See https://stackoverflow.com/questions/6061537/usage-of-maven-basedir-in-multi-module-setup but basically getting a path to a file that is accessed from multiple modules is a pain. It appears we need to choose between: 1) copying the code from findSupportedPython into the root pom. It's not a lot of code, but it's probably too much. 2) use a workaround that sets a per-project variable - https://stackoverflow.com/questions/1012402/maven2-property-that-indicates-the-parent-directory I'm open to other suggestions - for now I'm going to try out #2 to see if that works the way we want it to. > Make python maven variables available elsewhere > --- > > Key: BEAM-2460 > URL: https://issues.apache.org/jira/browse/BEAM-2460 > Project: Beam > Issue Type: Task > Components: sdk-java-core, sdk-py >Reporter: Stephen Sisk >Assignee: Stephen Sisk >Priority: Minor > > We're planning to start using perfkit to make it easy for users to run the IO > ITs. Perfkit is a python app, so we'll need to invoke the python interpreter. > findSupportedPython calculates this and sets the python.interpreter.bin > variable. It currently does so in the python project. > I believe the correct answer here is to move findSupportedPython so that's > it's invoked in the root pom. (but perhaps there's a more maven-y approach to > this problem?) > cc [~altay] [~markflyhigh] [~jasonkuster] [~davor] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (BEAM-2460) Make python maven variables available elsewhere
[ https://issues.apache.org/jira/browse/BEAM-2460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16053296#comment-16053296 ] Ahmet Altay commented on BEAM-2460: --- +1 to moving findSupportedPython to a root level if it will be used by other components. [~tibor.k...@gmail.com] is the author of the groovy file and at that we could not find another maven natural way of discovering python/pip binaries. > Make python maven variables available elsewhere > --- > > Key: BEAM-2460 > URL: https://issues.apache.org/jira/browse/BEAM-2460 > Project: Beam > Issue Type: Task > Components: sdk-java-core, sdk-py >Reporter: Stephen Sisk >Assignee: Stephen Sisk >Priority: Minor > > We're planning to start using perfkit to make it easy for users to run the IO > ITs. Perfkit is a python app, so we'll need to invoke the python interpreter. > findSupportedPython calculates this and sets the python.interpreter.bin > variable. It currently does so in the python project. > I believe the correct answer here is to move findSupportedPython so that's > it's invoked in the root pom. (but perhaps there's a more maven-y approach to > this problem?) > cc [~altay] [~markflyhigh] [~jasonkuster] [~davor] -- This message was sent by Atlassian JIRA (v6.4.14#64029)
