[ https://issues.apache.org/jira/browse/NIFI-4946?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16391553#comment-16391553 ]
ASF GitHub Bot commented on NIFI-4946: -------------------------------------- Github user zenfenan commented on a diff in the pull request: https://github.com/apache/nifi/pull/2521#discussion_r173216684 --- Diff: nifi-nar-bundles/nifi-spark-bundle/nifi-livy-processors/src/main/java/org/apache/nifi/processors/livy/ExecuteSparkInteractive.java --- @@ -83,6 +83,62 @@ .expressionLanguageSupported(true) .build(); + public static final PropertyDescriptor IS_BATCH_JOB = new PropertyDescriptor.Builder() + .name("exec-spark-iactive-is_batch_job") + .displayName("Is Batch Job") + .description("If true, the `Code` part is ignored and the flow file from previous stage is considered " + + "as a triggering event and not as code for Spark session. When `Wait` state is self routed" + + "the livy json response flow file from previous Spark job is used to poll the job status" + + "for sucess or failure") + .required(true) + .allowableValues("true", "false") + .defaultValue("false") + .build(); + + public static final PropertyDescriptor PY_FILES = new PropertyDescriptor.Builder() + .name("exec-spark-iactive-pyfiles") + .displayName("pyFiles") + .description("Python files to be used in this batch session that includes *.py, *.zip files") + .required(false) + .addValidator(StandardValidators.createURLorFileValidator()) + .expressionLanguageSupported(false) + .build(); + + public static final PropertyDescriptor JAR_FILES = new PropertyDescriptor.Builder() + .name("exec-spark-iactive-jarfiles") + .displayName("jars") + .description("jars to be used in this batch session") + .required(false) + .addValidator(StandardValidators.NON_EMPTY_VALIDATOR) + .expressionLanguageSupported(false) + .build(); + + public static final PropertyDescriptor NAME = new PropertyDescriptor.Builder() + .name("exec-spark-iactive-name") + .displayName("name") + .description("The name of this session") + .required(false) + .addValidator(StandardValidators.NON_EMPTY_VALIDATOR) + .expressionLanguageSupported(false) + .build(); + + public static final PropertyDescriptor ARGS = new PropertyDescriptor.Builder() + .name("exec-spark-iactive-args") + .displayName("args") + .description("The name of this session") + .required(false) + .addValidator(StandardValidators.NON_EMPTY_VALIDATOR) + .expressionLanguageSupported(false) + .build(); + + public static final PropertyDescriptor MAIN_PY_FILE = new PropertyDescriptor.Builder() + .name("exec-spark-iactive-main-py-file") + .displayName("file") --- End diff -- Same as the JARs case. Most of the `PropertyDescriptor` use all lowercase characters for `displayName`. Please change it. > nifi-spark-bundle : Adding support for pyfiles, file, jars options > ------------------------------------------------------------------ > > Key: NIFI-4946 > URL: https://issues.apache.org/jira/browse/NIFI-4946 > Project: Apache NiFi > Issue Type: New Feature > Components: Extensions > Affects Versions: 1.6.0 > Environment: Ubuntu 16.04, IntelliJ > Reporter: Mageswaran > Priority: Major > Fix For: 1.6.0 > > Attachments: nifi-spark-options.png, nifi-spark.png > > > Adding support for submitting PySpark based Sparks jobs (which is normally > structured as modules) over Livy on existing "ExecuteSparkInteractive" > processor. > This is done by reading file paths for pyfiles and file and an option from > user whether the processor should trigger a batch job or not. > [https://livy.incubator.apache.org/docs/latest/rest-api.html] > *Current Work flow Logic ( [https://github.com/apache/nifi/pull/2521 > )|https://github.com/apache/nifi/pull/2521]* > * Check whether the processor has to handle code or submit a Spark job > * Read incoming flow file > ** If batch == true > *** If flow file matches Livy `batches` JSON response through `wait` loop > **** Wait for Status Check Interval > **** Read the state > **** If state is `running` route it to `wait` or if it is `success` or > `dead` route it accordingly > *** Else > **** Ignore the flow file > **** Trigger the Spark job over Livy `batches` endpoint > **** Read the state of the submitted job > **** If state is `running` route it to `wait` or if it is `success` or > `dead` route it accordingly > ** Else: > *** Existing Logic to handle `Code` > > !nifi-spark-options.png! > !nifi-spark.png! > > Thanks. -- This message was sent by Atlassian JIRA (v7.6.3#76005)